{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Working with Bag of Words\n",
    "\n",
    "-------------------------------------\n",
    "\n",
    "In this example, we will download and preprocess the ham/spam text data.  We will then use a one-hot-encoding to make a bag of words set of features to use in logistic regression.\n",
    "\n",
    "We will use these one-hot-vectors for logistic regression to predict if a text is spam or ham.\n",
    "\n",
    "We start by loading the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import numpy as np\n",
    "import csv\n",
    "import string\n",
    "import requests\n",
    "import io\n",
    "from zipfile import ZipFile\n",
    "from tensorflow.contrib import learn\n",
    "from tensorflow.python.framework import ops\n",
    "ops.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We start a computation graph session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Start a graph session\n",
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Check if data was downloaded, otherwise download it and save for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "save_file_name = os.path.join('temp','temp_spam_data.csv')\n",
    "\n",
    "# Create directory if it doesn't exist\n",
    "if not os.path.exists('temp'):\n",
    "    os.makedirs('temp')\n",
    "\n",
    "if os.path.isfile(save_file_name):\n",
    "    text_data = []\n",
    "    with open(save_file_name, 'r') as temp_output_file:\n",
    "        reader = csv.reader(temp_output_file)\n",
    "        for row in reader:\n",
    "            text_data.append(row)\n",
    "else:\n",
    "    zip_url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip'\n",
    "    r = requests.get(zip_url)\n",
    "    z = ZipFile(io.BytesIO(r.content))\n",
    "    file = z.read('SMSSpamCollection')\n",
    "    # Format Data\n",
    "    text_data = file.decode()\n",
    "    text_data = text_data.encode('ascii',errors='ignore')\n",
    "    text_data = text_data.decode().split('\\n')\n",
    "    text_data = [x.split('\\t') for x in text_data if len(x)>=1]\n",
    "    \n",
    "    # And write to csv\n",
    "    with open(save_file_name, 'w') as temp_output_file:\n",
    "        writer = csv.writer(temp_output_file)\n",
    "        writer.writerows(text_data)\n",
    "\n",
    "texts = [x[1] for x in text_data]\n",
    "target = [x[0] for x in text_data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To reduce the potential vocabulary size, we normalize the text. To do this, we remove the influence of capitalization and numbers in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Relabel 'spam' as 1, 'ham' as 0\n",
    "target = [1 if x=='spam' else 0 for x in target]\n",
    "\n",
    "# Normalize text\n",
    "# Lower case\n",
    "texts = [x.lower() for x in texts]\n",
    "\n",
    "# Remove punctuation\n",
    "texts = [''.join(c for c in x if c not in string.punctuation) for x in texts]\n",
    "\n",
    "# Remove numbers\n",
    "texts = [''.join(c for c in x if c not in '0123456789') for x in texts]\n",
    "\n",
    "# Trim extra whitespace\n",
    "texts = [' '.join(x.split()) for x in texts]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To determine a good sentence length to pad/crop at, we plot a histogram of text lengths (in words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGltJREFUeJzt3X+UXGWd5/H3hwTklxIIbQaTaOMQQcYdEHsxDq4HE53l\nh5DMrnJgVYKT2exZcUYHXc14ZkRn1jlh1hFhnWFORpTgMEBE2OQI6042hEF2AO0A8puhxcSkzY82\nkmDEX5Hv/nG/DZemO13VXZVKP/15nVOnnvvc59Z9nkr1p26eulVXEYGZmZXrgE53wMzM2stBb2ZW\nOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAf9BCXpEUmnd7ofnSTp9yRtkrRb0hv30T6Pl/SApJ9I+qN9\nsc9R+nORpLvGsN3/lrSoHX2y/Y+Dfj8kaYOkdwype9EfdET8VkTcMcrjdEsKSVPb1NVO+xzwoYg4\nPCLu31tDSf2SDpE0T9LN49jnx4F1EfHyiLhyyD4ukPTYkLo1I9QtHUcfxi0izoyIFc1ul2+qg7fn\nJP2stvzesfZH0sH5Wp011sewkTnobcz2gzeQ1wCPjNZI0mxgR0T8DHgTcF+b9nkncIKkrtzvVOAk\n4JAhdW/Jtk2RNGVMPW6hfFM9PCIOB34AnFOru67T/bPhOegnqPpRv6RTJfVKekbSNkmfz2aDYbIz\nj7jeIukASX8qaaOk7ZKulXRE7XEvzHU7JP3ZkP18WtJNkv5B0jPARbnvuyXtlLRF0hclHVR7vJD0\nQUlP5nTHX0j6TUn/kv1dWW8/ZIzD9lXSyyTtBqYA35X0vVGerh5gfa2816CXdG5Oje2UdIek12f9\n7cDbgS/m8/m6+nYR0Q88Bbwtq06helP45yF1BwDfycd8fe5jZ+7z3Fo/rpF0laTbJP0UeLuk6ZJW\n53P3beA3a+0l6fJ8rp6R9JCkN4wwxjsk/UGWL5J0l6TPSXpa0vclnbn3p3TE525Kvm6ekvQjSddJ\nmpbrFkn6V0mH5fLvSdos6UheeK0+kc/tQkm/Iemb+dzsyOffxiIifNvPbsAG4B1D6i4C7hquDXA3\n8P4sHw7MzXI3EMDU2na/D/QBr822NwNfzXUnAruBtwIHUU2N/Kq2n0/n8kKqsDqE6gh5LjA19/cY\n8JHa/gJYBbwC+C3gF8Da3P8RwKPAohGehxH7Wnvs4/byPF4K7AR+Djyb5V8Du7I8ZZhtXgf8FHgn\ncCDVVE0fcFCuvwP4g73s8yvAFVn+GPDnwH8eUnd7lg/Mx/5kPt/zgJ8Ax+f6a7Kvp+XzfTBwA7AS\nOAx4A9A/+LoA/j3VG9o0QMDrgWNG6Ofz46B6bf0q+zkF+K/ADwGN4XX6CeBbwKuyv9cAX6mt/zrw\nd8AMYBvwzqw/OP89Z9XaXg5cka+tg4C3dfpvc6LeOt4B34b5R6n+gHZnGA3enmXkoL8T+Axw9JDH\n6ealQb8W+GBt+fj8I58KfAq4vrbuUOCXvDjo7xyl7x8BbqktB3BabXk98Ina8l8DXxjhsUbsa+2x\nRwz6bDOV6s1nBvA7wK2jtP8zYGVt+YAM09Nz+fmAHGH7i4D7s7yK6g3jhCF1l2b53wFbgQNq218P\nfDrL1wDX1tZNyfGfUKv7S14I+nnAv1K98R4wyjifH0f2uW/Iv3sAv9HA63Ro0H9/yL/3sfnaVS5P\np3oTeZh888v64YL+r4CvAa/t9N/kRL956mb/tTAipg3egA/upe1iqiPRxyV9R9K79tL2VcDG2vJG\nqjCckes2Da6IiGeBHUO231RfkPQ6Sd+QtDWnc/4SOHrINttq5Z8Ns3z4GPq6V5JOlrQTeBo4DngC\nWAecnlMB/6GRfUbEc1RjnjnaPtOdwG/ndMRc4O6IeBw4JuveygvTFK8CNuU+6mOs76v+fHdRjX/T\nkPaDfb0d+CLwN8B2ScslvaLBfm+tPc6zWRzp32VYkgTMBm7L53gncD/Vm+X0fOwdwC1U/3v8/EiP\nlT5L9aawTlKfpEua6Y+9wEFfgIh4MiIuAF4JXAbclPOgw/006Q+pPlAc9GpgD1X4bgGeP+tB0iHk\nH2h9d0OWrwIeB+ZExCuopiE09tE03Ne9iogH8g3ys8CnsvwocFK+eY505s2L9lkLr/5GOhwRT+Vj\nLAF+EBG7c9XdWXc4cE9tX7Ml1f8OXz1kX/Xne4Bq/LOHtK/v/8qIeBNVkL4O+G+N9LsVojoM7wfm\n1Q9SIuLgiPgRVJ8nARdQHanXz1p6yWs1InZFxIcj4jXAfwT+VNJp7R9JeRz0BZD0PkldeWS4M6uf\nowqG56jmuAddD/yxpGMlHU51BH5jROwBbgLOkfQ7+QHppxk9tF8OPAPslnQC1fxuq+ytr416E3Bf\njudVEdE3SvuVwNmS5ks6EPgo1ecK/9LEPr8FXJL3g+7Kut6ozv4BuJdqWuPjkg5U9b2Ic6jm4V8i\nIn5N9TnFpyUdKulE4Plz4SX9W0lvzn7/lOqzieeGe6w2+jtgmaoznZD0SknnZPlQ4B+ontOLgOMl\n/T5ARPyC6vOI51+r+aH4a/PNdhfV5yv7ejxFcNCX4QzgkTwT5Qrg/Ij4Wf4X/LPA/8v/Ss8Fvgx8\nlWr64PtUYfCHABHxSJZvoDq63w1spwq6kXwM+E9UHyL+PXBjC8c1Yl+bMHg65b+hmhfeq4h4Angf\n8D+BH1EF7zkR8csm9vnPVP+7qn+R6VtZ9/xplfmY5wBn5r7+Frgwp3pG8iGq/xVsJT/orK17BdW/\nwdNUUzo7gP/RRL9b4a+A/wvcLuknVG+Qp+S6vwYejYiv5Jvd+4HPSerO9Z8Cvpav1XOpPkxeR/Xa\nuhP4XETcvc9GUpDBD0jMXiKPondSTct8v9P9MbOx8RG9vYikc3Ja4DCq0ysfojq7wswmKAe9DbWA\n6kPCHwJzqKaB/N8+swnMUzdmZoXzEb2ZWeE6/aNUABx99NHR3d3d6W6YmU0o69ev/1FEdI3Wbr8I\n+u7ubnp7ezvdDTOzCUXSxtFbeerGzKx4Dnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCueg\nNzMrnIPezKxw+8U3Y0vTvfTWptpvWHZ2m3piZuYjejOz4jUU9JL+WNIjkh6WdL2kg/M6nvfm1dlv\nzGtyIulludyX67vbOQAzM9u7UYNe0kzgj4CeiHgDMAU4H7gMuDwijqO6RuXi3GQx8HTWX57tzMys\nQxqdupkKHCJpKnAo1YWj5wE35foVwMIsL8hlcv38vIq7mZl1wKhBHxH9VNcO/QFVwO8C1gM7I2JP\nNtsMzMzyTGBTbrsn208f+riSlkjqldQ7MDAw3nGYmdkIGpm6OZLqKP1Y4FXAYcAZ491xRCyPiJ6I\n6OnqGvV3883MbIwambp5B/D9iBiIiF8BNwOnAdNyKgdgFtCf5X5gNkCuPwLY0dJem5lZwxoJ+h8A\ncyUdmnPt84FHgXXAu7PNImBVllfnMrn+9vAVyM3MOqaROfp7qT5UvQ94KLdZDnwCuERSH9Uc/NW5\nydXA9Ky/BFjahn6bmVmDGvpmbERcClw6pPop4NRh2v4ceM/4u2ZmZq3gb8aamRXOQW9mVjgHvZlZ\n4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9mVjgHvZlZ4Rz0ZmaFc9CbmRXOQW9m\nVjgHvZlZ4Rz0ZmaFa+Ti4MdLeqB2e0bSRyQdJWmNpCfz/shsL0lXSuqT9KCkU9o/DDMzG0kjlxJ8\nIiJOjoiTgTcBzwK3UF0icG1EzAHW8sIlA88E5uRtCXBVOzpuZmaNaXbqZj7wvYjYCCwAVmT9CmBh\nlhcA10blHmCapGNa0lszM2tas0F/PnB9lmdExJYsbwVmZHkmsKm2zeasexFJSyT1SuodGBhoshtm\nZtaohoNe0kHAucDXhq6LiACimR1HxPKI6ImInq6urmY2NTOzJjRzRH8mcF9EbMvlbYNTMnm/Pev7\ngdm17WZlnZmZdUAzQX8BL0zbAKwGFmV5EbCqVn9hnn0zF9hVm+IxM7N9bGojjSQdBrwT+C+16mXA\nSkmLgY3AeVl/G3AW0Ed1hs4HWtZbMzNrWkNBHxE/BaYPqdtBdRbO0LYBXNyS3pmZ2bj5m7FmZoVz\n0JuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWOAe9mVnhHPRmZoVr6Jux1l7dS29tqv2GZWe3\nqSdmViIf0ZuZFc5Bb2ZWOAe9mVnhHPRmZoVz0JuZFc5Bb2ZWuIaCXtI0STdJelzSY5LeIukoSWsk\nPZn3R2ZbSbpSUp+kByWd0t4hmJnZ3jR6RH8F8M2IOAE4CXgMWAqsjYg5wNpchuoi4nPytgS4qqU9\nNjOzpowa9JKOAN4GXA0QEb+MiJ3AAmBFNlsBLMzyAuDaqNwDTJN0TMt7bmZmDWnkiP5YYAD4iqT7\nJX0pLxY+IyK2ZJutwIwszwQ21bbfnHUvImmJpF5JvQMDA2MfgZmZ7VUjQT8VOAW4KiLeCPyUF6Zp\ngOcvCB7N7DgilkdET0T0dHV1NbOpmZk1oZGg3wxsjoh7c/kmquDfNjglk/fbc30/MLu2/aysMzOz\nDhj1R80iYqukTZKOj4gngPnAo3lbBCzL+1W5yWrgQ5JuAN4M7KpN8UxIzf7omJnZ/qTRX6/8Q+A6\nSQcBTwEfoPrfwEpJi4GNwHnZ9jbgLKAPeDbbmplZhzQU9BHxANAzzKr5w7QN4OJx9svMzFrE34w1\nMyucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjnozcwK56A3Myucg97MrHAOejOzwjno\nzcwK56A3Myucg97MrHAOejOzwjnozcwK56A3MytcQ0EvaYOkhyQ9IKk3646StEbSk3l/ZNZL0pWS\n+iQ9KOmUdg7AzMz2rpkj+rdHxMkRMXhJwaXA2oiYA6zNZYAzgTl5WwJc1arOmplZ88YzdbMAWJHl\nFcDCWv21UbkHmCbpmHHsx8zMxqHRoA/gnyStl7Qk62ZExJYsbwVmZHkmsKm27easexFJSyT1Suod\nGBgYQ9fNzKwRUxts99aI6Jf0SmCNpMfrKyMiJEUzO46I5cBygJ6enqa2NTOzxjV0RB8R/Xm/HbgF\nOBXYNjglk/fbs3k/MLu2+aysMzOzDhg16CUdJunlg2Xgd4GHgdXAomy2CFiV5dXAhXn2zVxgV22K\nx8zM9rFGpm5mALdIGmz/jxHxTUnfAVZKWgxsBM7L9rcBZwF9wLPAB1reazMza9ioQR8RTwEnDVO/\nA5g/TH0AF7ekdzas7qW3NtV+w7Kz29QTM5sI/M1YM7PCOejNzArnoDczK5yD3syscA56M7PCOejN\nzArnoDczK5yD3syscA56M7PCOejNzArnoDczK5yD3syscA56M7PCOejNzArnoDczK5yD3syscA0H\nvaQpku6X9I1cPlbSvZL6JN0o6aCsf1ku9+X67vZ03czMGtHMEf2Hgcdqy5cBl0fEccDTwOKsXww8\nnfWXZzszM+uQhoJe0izgbOBLuSxgHnBTNlkBLMzyglwm18/P9mZm1gGNHtF/Afg48FwuTwd2RsSe\nXN4MzMzyTGATQK7fle1fRNISSb2SegcGBsbYfTMzG82oQS/pXcD2iFjfyh1HxPKI6ImInq6urlY+\ntJmZ1UxtoM1pwLmSzgIOBl4BXAFMkzQ1j9pnAf3Zvh+YDWyWNBU4AtjR8p6bmVlDRj2ij4g/iYhZ\nEdENnA/cHhHvBdYB785mi4BVWV6dy+T62yMiWtprMzNr2HjOo/8EcImkPqo5+Kuz/mpgetZfAiwd\nXxfNzGw8Gpm6eV5E3AHckeWngFOHafNz4D0t6JuZmbWAvxlrZla4po7obWLqXnpr09tsWHZ2G3pi\nZp3gI3ozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCuegNzMrnIPezKxwDnozs8I56M3MCueg\nNzMrnH/rxobV7O/j+LdxzPZfPqI3Myucg97MrHCNXBz8YEnflvRdSY9I+kzWHyvpXkl9km6UdFDW\nvyyX+3J9d3uHYGZme9PIEf0vgHkRcRJwMnCGpLnAZcDlEXEc8DSwONsvBp7O+suznZmZdUgjFweP\niNidiwfmLYB5wE1ZvwJYmOUFuUyuny9JLeuxmZk1paE5eklTJD0AbAfWAN8DdkbEnmyyGZiZ5ZnA\nJoBcv4vq4uFDH3OJpF5JvQMDA+MbhZmZjaihoI+IX0fEycAsqguCnzDeHUfE8ojoiYierq6u8T6c\nmZmNoKmzbiJiJ7AOeAswTdLgefizgP4s9wOzAXL9EcCOlvTWzMyaNuoXpiR1Ab+KiJ2SDgHeSfUB\n6zrg3cANwCJgVW6yOpfvzvW3R0S0oe+2H9nfvmC1v/XHrJMa+WbsMcAKSVOo/gewMiK+IelR4AZJ\n/x24H7g6218NfFVSH/Bj4Pw29NvMzBo0atBHxIPAG4epf4pqvn5o/c+B97Skd2ap2SN0M3uBvxlr\nZlY4B72ZWeH865XWEZ6KMdt3fERvZlY4B72ZWeE8dWPG2KaSfO69TRQOerMx8peybKLw1I2ZWeEc\n9GZmhXPQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZlY4B72ZWeFGDXpJsyWtk/SopEckfTjr\nj5K0RtKTeX9k1kvSlZL6JD0o6ZR2D8LMzEbWyBH9HuCjEXEiMBe4WNKJwFJgbUTMAdbmMsCZwJy8\nLQGuanmvzcysYaMGfURsiYj7svwT4DFgJrAAWJHNVgALs7wAuDYq9wDTJB3T8p6bmVlDmpqjl9RN\ndf3Ye4EZEbElV20FZmR5JrCpttnmrBv6WEsk9UrqHRgYaLLbZmbWqIaDXtLhwNeBj0TEM/V1ERFA\nNLPjiFgeET0R0dPV1dXMpmZm1oSGgl7SgVQhf11E3JzV2wanZPJ+e9b3A7Nrm8/KOjMz64BGzroR\ncDXwWER8vrZqNbAoy4uAVbX6C/Psm7nArtoUj5mZ7WONXHjkNOD9wEOSHsi6TwLLgJWSFgMbgfNy\n3W3AWUAf8CzwgZb22MzMmjJq0EfEXYBGWD1/mPYBXDzOfpmZWYv4m7FmZoVz0JuZFc5Bb2ZWOAe9\nmVnhHPRmZoVz0JuZFc5Bb2ZWuEa+MGVmLdC99Nam2m9YdnabemKTjY/ozcwK56A3Myucg97MrHAO\nejOzwk26D2Ob/UDMzGyi8xG9mVnhJt0RvdlE4dMxrVV8RG9mVrhGLiX4ZUnbJT1cqztK0hpJT+b9\nkVkvSVdK6pP0oKRT2tl5MzMbXSNH9NcAZwypWwqsjYg5wNpcBjgTmJO3JcBVremmmZmN1ahBHxF3\nAj8eUr0AWJHlFcDCWv21UbkHmCbpmFZ11szMmjfWOfoZEbEly1uBGVmeCWyqtducdWZm1iHj/jA2\nLwYezW4naYmkXkm9AwMD4+2GmZmNYKxBv21wSibvt2d9PzC71m5W1r1ERCyPiJ6I6Onq6hpjN8zM\nbDRjDfrVwKIsLwJW1eovzLNv5gK7alM8ZmbWAaN+YUrS9cDpwNGSNgOXAsuAlZIWAxuB87L5bcBZ\nQB/wLPCBNvTZzMyaMGrQR8QFI6yaP0zbAC4eb6fMzKx1/BMIZoXwTybYSPwTCGZmhXPQm5kVzkFv\nZlY4B72ZWeEc9GZmhXPQm5kVzkFvZla4CX8evS/2bWa2dxM+6M1sbPwFq8nDUzdmZoVz0JuZFc5T\nN2bWkLF8Hubpnv2Dj+jNzArnoDczK5yD3syscJ6jN7O28Smc+4e2BL2kM4ArgCnAlyJiWTv2Y2Zl\nafcXICfrG0nLg17SFOBvgHcCm4HvSFodEY+2el9mZu20L755vy/efNpxRH8q0BcRTwFIugFYADjo\nzayjJutPprQj6GcCm2rLm4E3D20kaQmwJBd3S3pijPs7GvjRGLedyCbruGHyjt3jLpAuG3FVI+N+\nTSP76NiHsRGxHFg+3seR1BsRPS3o0oQyWccNk3fsHvfk0spxt+P0yn5gdm15VtaZmVkHtCPovwPM\nkXSspIOA84HVbdiPmZk1oOVTNxGxR9KHgP9DdXrllyPikVbvp2bc0z8T1GQdN0zesXvck0vLxq2I\naNVjmZnZfsg/gWBmVjgHvZlZ4SZ00Es6Q9ITkvokLe10f9pF0pclbZf0cK3uKElrJD2Z90d2so/t\nIGm2pHWSHpX0iKQPZ33RY5d0sKRvS/pujvszWX+spHvz9X5jnuxQHElTJN0v6Ru5XPy4JW2Q9JCk\nByT1Zl3LXucTNuhrP7VwJnAicIGkEzvbq7a5BjhjSN1SYG1EzAHW5nJp9gAfjYgTgbnAxflvXPrY\nfwHMi4iTgJOBMyTNBS4DLo+I44CngcUd7GM7fRh4rLY8Wcb99og4uXbufMte5xM26Kn91EJE/BIY\n/KmF4kTEncCPh1QvAFZkeQWwcJ92ah+IiC0RcV+Wf0L1xz+Twsceld25eGDeApgH3JT1xY0bQNIs\n4GzgS7ksJsG4R9Cy1/lEDvrhfmphZof60gkzImJLlrcCMzrZmXaT1A28EbiXSTD2nL54ANgOrAG+\nB+yMiD3ZpNTX+xeAjwPP5fJ0Jse4A/gnSevz52Ggha9z/x59ASIiJBV7nqykw4GvAx+JiGeqg7xK\nqWOPiF8DJ0uaBtwCnNDhLrWdpHcB2yNivaTTO92ffeytEdEv6ZXAGkmP11eO93U+kY/oJ/tPLWyT\ndAxA3m/vcH/aQtKBVCF/XUTcnNWTYuwAEbETWAe8BZgmafDgrMTX+2nAuZI2UE3FzqO6rkXp4yYi\n+vN+O9Ub+6m08HU+kYN+sv/UwmpgUZYXAas62Je2yPnZq4HHIuLztVVFj11SVx7JI+kQqms7PEYV\n+O/OZsWNOyL+JCJmRUQ31d/z7RHxXgoft6TDJL18sAz8LvAwLXydT+hvxko6i2pOb/CnFj7b4S61\nhaTrgdOpfrZ0G3Ap8L+AlcCrgY3AeREx9APbCU3SW4FvAQ/xwpztJ6nm6Ysdu6TfpvrwbQrVwdjK\niPhzSa+lOtI9CrgfeF9E/KJzPW2fnLr5WES8q/Rx5/huycWpwD9GxGclTadFr/MJHfRmZja6iTx1\nY2ZmDXDQm5kVzkFvZlY4B72ZWeEc9GZmhXPQm5kVzkFvZla4/w/awuSlohogaQAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5d3a34ecf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# Plot histogram of text lengths\n",
    "text_lengths = [len(x.split()) for x in texts]\n",
    "text_lengths = [x for x in text_lengths if x < 50]\n",
    "plt.hist(text_lengths, bins=25)\n",
    "plt.title('Histogram of # of Words in Texts')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We crop/pad all texts to be 25 words long.  We also will filter out any words that do not appear at least 3 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Choose max text word length at 25\n",
    "sentence_size = 25\n",
    "min_word_freq = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "TensorFlow has a built in text processing function called `VocabularyProcessor()`. We use this function to process the texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Setup vocabulary processor\n",
    "vocab_processor = learn.preprocessing.VocabularyProcessor(sentence_size, min_frequency=min_word_freq)\n",
    "\n",
    "# Have to fit transform to get length of unique words.\n",
    "vocab_processor.transform(texts)\n",
    "transformed_texts = np.array([x for x in vocab_processor.transform(texts)])\n",
    "embedding_size = len(np.unique(transformed_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To test our logistic model (predicting spam/ham), we split the texts into a train and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Split up data set into train/test\n",
    "train_indices = np.random.choice(len(texts), round(len(texts)*0.8), replace=False)\n",
    "test_indices = np.array(list(set(range(len(texts))) - set(train_indices)))\n",
    "texts_train = [x for ix, x in enumerate(texts) if ix in train_indices]\n",
    "texts_test = [x for ix, x in enumerate(texts) if ix in test_indices]\n",
    "target_train = [x for ix, x in enumerate(target) if ix in train_indices]\n",
    "target_test = [x for ix, x in enumerate(target) if ix in test_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For one-hot-encoding, we setup an identity matrix for the TensorFlow embedding lookup.\n",
    "\n",
    "We also create the variables and placeholders for the logistic regression we will perform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Setup Index Matrix for one-hot-encoding\n",
    "identity_mat = tf.diag(tf.ones(shape=[embedding_size]))\n",
    "\n",
    "# Create variables for logistic regression\n",
    "A = tf.Variable(tf.random_normal(shape=[embedding_size,1]))\n",
    "b = tf.Variable(tf.random_normal(shape=[1,1]))\n",
    "\n",
    "# Initialize placeholders\n",
    "x_data = tf.placeholder(shape=[sentence_size], dtype=tf.int32)\n",
    "y_target = tf.placeholder(shape=[1, 1], dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next, we create the text-word embedding lookup with the prior identity matrix.\n",
    "\n",
    "Our logistic regression will use the counts of the words as the input.  The counts are created by summing the embedding output across the rows.\n",
    "\n",
    "Then we declare the logistic regression operations. Note that we do not wrap the logistic operations in the sigmoid function because this will be done in the loss function later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Text-Vocab Embedding\n",
    "x_embed = tf.nn.embedding_lookup(identity_mat, x_data)\n",
    "x_col_sums = tf.reduce_sum(x_embed, 0)\n",
    "\n",
    "# Declare model operations\n",
    "x_col_sums_2D = tf.expand_dims(x_col_sums, 0)\n",
    "model_output = tf.add(tf.matmul(x_col_sums_2D, A), b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we declare our loss function (which has the sigmoid built in), prediction operations, optimizer, and initialize the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Declare loss function (Cross Entropy loss)\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=model_output, labels=y_target))\n",
    "\n",
    "# Prediction operation\n",
    "prediction = tf.sigmoid(model_output)\n",
    "\n",
    "# Declare optimizer\n",
    "my_opt = tf.train.GradientDescentOptimizer(0.001)\n",
    "train_step = my_opt.minimize(loss)\n",
    "\n",
    "# Intitialize Variables\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we loop through the iterations and fit the logistic regression on wether or not the text is spam or ham."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Training Over 4459 Sentences.\n",
      "Training Observation #10: Loss = 10.2778\n",
      "Training Observation #20: Loss = 1.19316e-09\n",
      "Training Observation #30: Loss = 3.12619e-06\n",
      "Training Observation #40: Loss = 8.27559e-10\n",
      "Training Observation #50: Loss = 10.9895\n",
      "Training Observation #60: Loss = 3.89441e-05\n",
      "Training Observation #70: Loss = 8.85592e-07\n",
      "Training Observation #80: Loss = 0.790782\n",
      "Training Observation #90: Loss = 1.02829e-06\n",
      "Training Observation #100: Loss = 3.81317\n",
      "Training Observation #110: Loss = 4.6235e-09\n",
      "Training Observation #120: Loss = 1.44432e-05\n",
      "Training Observation #130: Loss = 1.5916e-08\n",
      "Training Observation #140: Loss = 0.000100448\n",
      "Training Observation #150: Loss = 1.17368e-05\n",
      "Training Observation #160: Loss = 7.48297e-09\n",
      "Training Observation #170: Loss = 1.29962e-05\n",
      "Training Observation #180: Loss = 2.43554e-08\n",
      "Training Observation #190: Loss = 1.3747e-06\n",
      "Training Observation #200: Loss = 3.06648e-10\n",
      "Training Observation #210: Loss = 0.0520885\n",
      "Training Observation #220: Loss = 0.00441677\n",
      "Training Observation #230: Loss = 1.38112e-06\n",
      "Training Observation #240: Loss = 3.68009e-05\n",
      "Training Observation #250: Loss = 0.0068205\n",
      "Training Observation #260: Loss = 0.000106576\n",
      "Training Observation #270: Loss = 1.30045e-08\n",
      "Training Observation #280: Loss = 3.05209e-08\n",
      "Training Observation #290: Loss = 3.18688e-06\n",
      "Training Observation #300: Loss = 1.66265e-06\n",
      "Training Observation #310: Loss = 0.350584\n",
      "Training Observation #320: Loss = 1.91021e-05\n",
      "Training Observation #330: Loss = 2.51232e-06\n",
      "Training Observation #340: Loss = 3.14701e-06\n",
      "Training Observation #350: Loss = 7.29681e-08\n",
      "Training Observation #360: Loss = 0.980764\n",
      "Training Observation #370: Loss = 0.0190124\n",
      "Training Observation #380: Loss = 1.81225e-05\n",
      "Training Observation #390: Loss = 6.42406e-06\n",
      "Training Observation #400: Loss = 2.0121e-06\n",
      "Training Observation #410: Loss = 0.908233\n",
      "Training Observation #420: Loss = 1.79508e-05\n",
      "Training Observation #430: Loss = 0.00689403\n",
      "Training Observation #440: Loss = 0.000131868\n",
      "Training Observation #450: Loss = 6.90337\n",
      "Training Observation #460: Loss = 0.00360844\n",
      "Training Observation #470: Loss = 6.17646\n",
      "Training Observation #480: Loss = 13.5849\n",
      "Training Observation #490: Loss = 1.0187\n",
      "Training Observation #500: Loss = 1.20479e-05\n",
      "Training Observation #510: Loss = 9.7865\n",
      "Training Observation #520: Loss = 2.09577e-05\n",
      "Training Observation #530: Loss = 0.00413652\n",
      "Training Observation #540: Loss = 0.00154006\n",
      "Training Observation #550: Loss = 8.37956e-07\n",
      "Training Observation #560: Loss = 2.06303e-05\n",
      "Training Observation #570: Loss = 7.73394\n",
      "Training Observation #580: Loss = 3.6019e-06\n",
      "Training Observation #590: Loss = 0.00272255\n",
      "Training Observation #600: Loss = 11.327\n",
      "Training Observation #610: Loss = 8.39739e-05\n",
      "Training Observation #620: Loss = 0.0154216\n",
      "Training Observation #630: Loss = 0.168612\n",
      "Training Observation #640: Loss = 2.38635e-06\n",
      "Training Observation #650: Loss = 7.84515e-06\n",
      "Training Observation #660: Loss = 1.6177e-05\n",
      "Training Observation #670: Loss = 0.473604\n",
      "Training Observation #680: Loss = 0.268214\n",
      "Training Observation #690: Loss = 2.23086\n",
      "Training Observation #700: Loss = 2.72788\n",
      "Training Observation #710: Loss = 3.46453e-06\n",
      "Training Observation #720: Loss = 9.34344e-06\n",
      "Training Observation #730: Loss = 1.63389\n",
      "Training Observation #740: Loss = 0.0215835\n",
      "Training Observation #750: Loss = 0.00011134\n",
      "Training Observation #760: Loss = 0.00539058\n",
      "Training Observation #770: Loss = 0.0643511\n",
      "Training Observation #780: Loss = 0.00192186\n",
      "Training Observation #790: Loss = 0.47168\n",
      "Training Observation #800: Loss = 0.460034\n",
      "Training Observation #810: Loss = 0.000319884\n",
      "Training Observation #820: Loss = 0.000138022\n",
      "Training Observation #830: Loss = 0.00016684\n",
      "Training Observation #840: Loss = 0.000915529\n",
      "Training Observation #850: Loss = 0.0755449\n",
      "Training Observation #860: Loss = 0.000462116\n",
      "Training Observation #870: Loss = 0.00070938\n",
      "Training Observation #880: Loss = 0.00288585\n",
      "Training Observation #890: Loss = 9.1019\n",
      "Training Observation #900: Loss = 2.44433\n",
      "Training Observation #910: Loss = 4.52815\n",
      "Training Observation #920: Loss = 0.000285115\n",
      "Training Observation #930: Loss = 0.00136862\n",
      "Training Observation #940: Loss = 2.80622e-06\n",
      "Training Observation #950: Loss = 0.000840566\n",
      "Training Observation #960: Loss = 0.896023\n",
      "Training Observation #970: Loss = 0.000246887\n",
      "Training Observation #980: Loss = 0.00130488\n",
      "Training Observation #990: Loss = 3.13126\n",
      "Training Observation #1000: Loss = 1.8586\n",
      "Training Observation #1010: Loss = 7.10204e-07\n",
      "Training Observation #1020: Loss = 0.0547286\n",
      "Training Observation #1030: Loss = 0.00589213\n",
      "Training Observation #1040: Loss = 7.23177e-05\n",
      "Training Observation #1050: Loss = 0.319899\n",
      "Training Observation #1060: Loss = 0.00114894\n",
      "Training Observation #1070: Loss = 0.00259651\n",
      "Training Observation #1080: Loss = 0.0179473\n",
      "Training Observation #1090: Loss = 2.35176\n",
      "Training Observation #1100: Loss = 0.00164634\n",
      "Training Observation #1110: Loss = 0.52194\n",
      "Training Observation #1120: Loss = 6.61559e-07\n",
      "Training Observation #1130: Loss = 0.65213\n",
      "Training Observation #1140: Loss = 5.91898\n",
      "Training Observation #1150: Loss = 9.04454e-05\n",
      "Training Observation #1160: Loss = 0.00014283\n",
      "Training Observation #1170: Loss = 3.48781e-05\n",
      "Training Observation #1180: Loss = 12.4596\n",
      "Training Observation #1190: Loss = 0.00101821\n",
      "Training Observation #1200: Loss = 0.0321773\n",
      "Training Observation #1210: Loss = 0.00263232\n",
      "Training Observation #1220: Loss = 0.00268328\n",
      "Training Observation #1230: Loss = 2.77804e-05\n",
      "Training Observation #1240: Loss = 9.40998e-05\n",
      "Training Observation #1250: Loss = 0.000179736\n",
      "Training Observation #1260: Loss = 0.141412\n",
      "Training Observation #1270: Loss = 0.00120529\n",
      "Training Observation #1280: Loss = 0.00177561\n",
      "Training Observation #1290: Loss = 0.000518835\n",
      "Training Observation #1300: Loss = 0.000263517\n",
      "Training Observation #1310: Loss = 0.00225621\n",
      "Training Observation #1320: Loss = 0.00146273\n",
      "Training Observation #1330: Loss = 0.000311388\n",
      "Training Observation #1340: Loss = 1.07662\n",
      "Training Observation #1350: Loss = 0.00544067\n",
      "Training Observation #1360: Loss = 0.000168739\n",
      "Training Observation #1370: Loss = 4.31152e-07\n",
      "Training Observation #1380: Loss = 0.0617295\n",
      "Training Observation #1390: Loss = 4.97479\n",
      "Training Observation #1400: Loss = 6.78335e-05\n",
      "Training Observation #1410: Loss = 0.000876943\n",
      "Training Observation #1420: Loss = 0.0218916\n",
      "Training Observation #1430: Loss = 0.0012828\n",
      "Training Observation #1440: Loss = 2.59641\n",
      "Training Observation #1450: Loss = 0.00974655\n",
      "Training Observation #1460: Loss = 4.19929e-05\n",
      "Training Observation #1470: Loss = 0.460421\n",
      "Training Observation #1480: Loss = 0.00121001\n",
      "Training Observation #1490: Loss = 0.000167905\n",
      "Training Observation #1500: Loss = 2.59278\n",
      "Training Observation #1510: Loss = 0.00945739\n",
      "Training Observation #1520: Loss = 5.85294\n",
      "Training Observation #1530: Loss = 0.00526647\n",
      "Training Observation #1540: Loss = 0.015951\n",
      "Training Observation #1550: Loss = 0.000975393\n",
      "Training Observation #1560: Loss = 0.000156549\n",
      "Training Observation #1570: Loss = 0.000189162\n",
      "Training Observation #1580: Loss = 0.000656738\n",
      "Training Observation #1590: Loss = 0.00116956\n",
      "Training Observation #1600: Loss = 0.161834\n",
      "Training Observation #1610: Loss = 0.00010154\n",
      "Training Observation #1620: Loss = 1.312\n",
      "Training Observation #1630: Loss = 0.000897444\n",
      "Training Observation #1640: Loss = 0.0117051\n",
      "Training Observation #1650: Loss = 0.00643426\n",
      "Training Observation #1660: Loss = 0.00840182\n",
      "Training Observation #1670: Loss = 0.00175345\n",
      "Training Observation #1680: Loss = 0.00358194\n",
      "Training Observation #1690: Loss = 0.00118526\n",
      "Training Observation #1700: Loss = 0.000355493\n",
      "Training Observation #1710: Loss = 0.000814216\n",
      "Training Observation #1720: Loss = 5.49319e-06\n",
      "Training Observation #1730: Loss = 0.00598321\n",
      "Training Observation #1740: Loss = 1.55247e-05\n",
      "Training Observation #1750: Loss = 0.000221107\n",
      "Training Observation #1760: Loss = 0.00338075\n",
      "Training Observation #1770: Loss = 0.0197768\n",
      "Training Observation #1780: Loss = 0.000547182\n",
      "Training Observation #1790: Loss = 8.42662\n",
      "Training Observation #1800: Loss = 0.466745\n",
      "Training Observation #1810: Loss = 0.00608892\n",
      "Training Observation #1820: Loss = 4.41871e-05\n",
      "Training Observation #1830: Loss = 8.69296e-06\n",
      "Training Observation #1840: Loss = 0.0326766\n",
      "Training Observation #1850: Loss = 0.0909576\n",
      "Training Observation #1860: Loss = 0.000568859\n",
      "Training Observation #1870: Loss = 0.0034418\n",
      "Training Observation #1880: Loss = 5.62818\n",
      "Training Observation #1890: Loss = 0.116249\n",
      "Training Observation #1900: Loss = 8.74811e-07\n",
      "Training Observation #1910: Loss = 7.37527\n",
      "Training Observation #1920: Loss = 0.086606\n",
      "Training Observation #1930: Loss = 0.00391311\n",
      "Training Observation #1940: Loss = 0.000845698\n",
      "Training Observation #1950: Loss = 1.58553\n",
      "Training Observation #1960: Loss = 0.00381753\n",
      "Training Observation #1970: Loss = 0.000634821\n",
      "Training Observation #1980: Loss = 0.00261\n",
      "Training Observation #1990: Loss = 5.74593\n",
      "Training Observation #2000: Loss = 1.53101\n",
      "Training Observation #2010: Loss = 0.573755\n",
      "Training Observation #2020: Loss = 0.091081\n",
      "Training Observation #2030: Loss = 10.1654\n",
      "Training Observation #2040: Loss = 2.99389e-05\n",
      "Training Observation #2050: Loss = 0.000196418\n",
      "Training Observation #2060: Loss = 2.68186e-05\n",
      "Training Observation #2070: Loss = 0.000167064\n",
      "Training Observation #2080: Loss = 0.106499\n",
      "Training Observation #2090: Loss = 0.00100722\n",
      "Training Observation #2100: Loss = 0.0367384\n",
      "Training Observation #2110: Loss = 4.96846\n",
      "Training Observation #2120: Loss = 0.000493067\n",
      "Training Observation #2130: Loss = 2.52678\n",
      "Training Observation #2140: Loss = 0.00853967\n",
      "Training Observation #2150: Loss = 2.28872\n",
      "Training Observation #2160: Loss = 1.62873\n",
      "Training Observation #2170: Loss = 0.0265156\n",
      "Training Observation #2180: Loss = 0.000398289\n",
      "Training Observation #2190: Loss = 0.100676\n",
      "Training Observation #2200: Loss = 0.00429453\n",
      "Training Observation #2210: Loss = 0.00315549\n",
      "Training Observation #2220: Loss = 4.77878e-06\n",
      "Training Observation #2230: Loss = 0.00078537\n",
      "Training Observation #2240: Loss = 5.26079\n",
      "Training Observation #2250: Loss = 0.000291106\n",
      "Training Observation #2260: Loss = 1.12961\n",
      "Training Observation #2270: Loss = 0.00676956\n",
      "Training Observation #2280: Loss = 0.00071802\n",
      "Training Observation #2290: Loss = 12.9432\n",
      "Training Observation #2300: Loss = 0.00873982\n",
      "Training Observation #2310: Loss = 0.0253997\n",
      "Training Observation #2320: Loss = 0.0142401\n",
      "Training Observation #2330: Loss = 0.000244211\n",
      "Training Observation #2340: Loss = 0.00895472\n",
      "Training Observation #2350: Loss = 6.17442e-05\n",
      "Training Observation #2360: Loss = 6.92446\n",
      "Training Observation #2370: Loss = 0.00184856\n",
      "Training Observation #2380: Loss = 2.08595\n",
      "Training Observation #2390: Loss = 0.000166794\n",
      "Training Observation #2400: Loss = 0.307666\n",
      "Training Observation #2410: Loss = 0.000338813\n",
      "Training Observation #2420: Loss = 0.000173173\n",
      "Training Observation #2430: Loss = 0.0171613\n",
      "Training Observation #2440: Loss = 0.00607599\n",
      "Training Observation #2450: Loss = 0.000109539\n",
      "Training Observation #2460: Loss = 0.000190355\n",
      "Training Observation #2470: Loss = 0.00212404\n",
      "Training Observation #2480: Loss = 0.00137951\n",
      "Training Observation #2490: Loss = 0.000163508\n",
      "Training Observation #2500: Loss = 0.00102085\n",
      "Training Observation #2510: Loss = 3.16698\n",
      "Training Observation #2520: Loss = 8.75264\n",
      "Training Observation #2530: Loss = 1.26042\n",
      "Training Observation #2540: Loss = 3.72369e-05\n",
      "Training Observation #2550: Loss = 4.23291\n",
      "Training Observation #2560: Loss = 0.000223255\n",
      "Training Observation #2570: Loss = 0.00222571\n",
      "Training Observation #2580: Loss = 4.78428e-05\n",
      "Training Observation #2590: Loss = 0.000395708\n",
      "Training Observation #2600: Loss = 5.40112e-05\n",
      "Training Observation #2610: Loss = 6.12541e-06\n",
      "Training Observation #2620: Loss = 7.23761\n",
      "Training Observation #2630: Loss = 6.79794\n",
      "Training Observation #2640: Loss = 0.00222946\n",
      "Training Observation #2650: Loss = 3.83829e-06\n",
      "Training Observation #2660: Loss = 7.0499e-05\n",
      "Training Observation #2670: Loss = 0.00158664\n",
      "Training Observation #2680: Loss = 0.000366441\n",
      "Training Observation #2690: Loss = 0.000109369\n",
      "Training Observation #2700: Loss = 0.000108413\n",
      "Training Observation #2710: Loss = 0.000272305\n",
      "Training Observation #2720: Loss = 0.00331319\n",
      "Training Observation #2730: Loss = 0.0042191\n",
      "Training Observation #2740: Loss = 0.0217124\n",
      "Training Observation #2750: Loss = 4.05261\n",
      "Training Observation #2760: Loss = 0.00545638\n",
      "Training Observation #2770: Loss = 0.0149828\n",
      "Training Observation #2780: Loss = 4.56443\n",
      "Training Observation #2790: Loss = 0.0229984\n",
      "Training Observation #2800: Loss = 0.128654\n",
      "Training Observation #2810: Loss = 0.00508549\n",
      "Training Observation #2820: Loss = 8.78744e-09\n",
      "Training Observation #2830: Loss = 2.53787\n",
      "Training Observation #2840: Loss = 0.000480627\n",
      "Training Observation #2850: Loss = 0.000901423\n",
      "Training Observation #2860: Loss = 1.8135\n",
      "Training Observation #2870: Loss = 5.72638\n",
      "Training Observation #2880: Loss = 1.13754\n",
      "Training Observation #2890: Loss = 0.00131672\n",
      "Training Observation #2900: Loss = 0.000396727\n",
      "Training Observation #2910: Loss = 0.0465008\n",
      "Training Observation #2920: Loss = 2.71821\n",
      "Training Observation #2930: Loss = 0.00630605\n",
      "Training Observation #2940: Loss = 0.00228251\n",
      "Training Observation #2950: Loss = 0.00184151\n",
      "Training Observation #2960: Loss = 5.81618e-05\n",
      "Training Observation #2970: Loss = 0.119603\n",
      "Training Observation #2980: Loss = 0.000120104\n",
      "Training Observation #2990: Loss = 0.00115399\n",
      "Training Observation #3000: Loss = 0.000410696\n",
      "Training Observation #3010: Loss = 0.000724803\n",
      "Training Observation #3020: Loss = 0.00712591\n",
      "Training Observation #3030: Loss = 0.00463644\n",
      "Training Observation #3040: Loss = 2.72483\n",
      "Training Observation #3050: Loss = 0.828409\n",
      "Training Observation #3060: Loss = 0.00067468\n",
      "Training Observation #3070: Loss = 0.0109851\n",
      "Training Observation #3080: Loss = 0.000334393\n",
      "Training Observation #3090: Loss = 0.312254\n",
      "Training Observation #3100: Loss = 2.61418\n",
      "Training Observation #3110: Loss = 9.4969e-06\n",
      "Training Observation #3120: Loss = 0.00038713\n",
      "Training Observation #3130: Loss = 0.00433318\n",
      "Training Observation #3140: Loss = 0.00464778\n",
      "Training Observation #3150: Loss = 3.87077\n",
      "Training Observation #3160: Loss = 0.0122973\n",
      "Training Observation #3170: Loss = 0.0285987\n",
      "Training Observation #3180: Loss = 0.00532439\n",
      "Training Observation #3190: Loss = 0.013977\n",
      "Training Observation #3200: Loss = 12.4325\n",
      "Training Observation #3210: Loss = 4.88311\n",
      "Training Observation #3220: Loss = 5.72166\n",
      "Training Observation #3230: Loss = 0.000307598\n",
      "Training Observation #3240: Loss = 0.00438079\n",
      "Training Observation #3250: Loss = 6.77368e-05\n",
      "Training Observation #3260: Loss = 0.00235564\n",
      "Training Observation #3270: Loss = 0.0558286\n",
      "Training Observation #3280: Loss = 5.39449\n",
      "Training Observation #3290: Loss = 0.021399\n",
      "Training Observation #3300: Loss = 0.00230528\n",
      "Training Observation #3310: Loss = 0.00141438\n",
      "Training Observation #3320: Loss = 0.00581662\n",
      "Training Observation #3330: Loss = 0.0183138\n",
      "Training Observation #3340: Loss = 0.00012653\n",
      "Training Observation #3350: Loss = 0.212888\n",
      "Training Observation #3360: Loss = 0.000196873\n",
      "Training Observation #3370: Loss = 0.000197204\n",
      "Training Observation #3380: Loss = 0.00108075\n",
      "Training Observation #3390: Loss = 12.307\n",
      "Training Observation #3400: Loss = 0.0085976\n",
      "Training Observation #3410: Loss = 2.27919\n",
      "Training Observation #3420: Loss = 0.370915\n",
      "Training Observation #3430: Loss = 0.00160115\n",
      "Training Observation #3440: Loss = 0.00179282\n",
      "Training Observation #3450: Loss = 2.79052\n",
      "Training Observation #3460: Loss = 6.41842\n",
      "Training Observation #3470: Loss = 0.586124\n",
      "Training Observation #3480: Loss = 0.902232\n",
      "Training Observation #3490: Loss = 1.97581e-05\n",
      "Training Observation #3500: Loss = 0.0240024\n",
      "Training Observation #3510: Loss = 1.37527e-05\n",
      "Training Observation #3520: Loss = 0.0024462\n",
      "Training Observation #3530: Loss = 3.43653\n",
      "Training Observation #3540: Loss = 0.00115513\n",
      "Training Observation #3550: Loss = 0.00575808\n",
      "Training Observation #3560: Loss = 0.22644\n",
      "Training Observation #3570: Loss = 0.000540085\n",
      "Training Observation #3580: Loss = 0.00082151\n",
      "Training Observation #3590: Loss = 4.31525e-05\n",
      "Training Observation #3600: Loss = 0.000302412\n",
      "Training Observation #3610: Loss = 0.857996\n",
      "Training Observation #3620: Loss = 0.01577\n",
      "Training Observation #3630: Loss = 0.0218573\n",
      "Training Observation #3640: Loss = 0.0114747\n",
      "Training Observation #3650: Loss = 0.0459154\n",
      "Training Observation #3660: Loss = 0.000377326\n",
      "Training Observation #3670: Loss = 0.0399131\n",
      "Training Observation #3680: Loss = 0.390027\n",
      "Training Observation #3690: Loss = 0.00379758\n",
      "Training Observation #3700: Loss = 0.0291584\n",
      "Training Observation #3710: Loss = 0.0816857\n",
      "Training Observation #3720: Loss = 0.000582221\n",
      "Training Observation #3730: Loss = 0.00199638\n",
      "Training Observation #3740: Loss = 0.0231141\n",
      "Training Observation #3750: Loss = 0.00029956\n",
      "Training Observation #3760: Loss = 3.37333\n",
      "Training Observation #3770: Loss = 0.087676\n",
      "Training Observation #3780: Loss = 0.000255297\n",
      "Training Observation #3790: Loss = 0.000127191\n",
      "Training Observation #3800: Loss = 0.00650525\n",
      "Training Observation #3810: Loss = 0.0873599\n",
      "Training Observation #3820: Loss = 0.000594171\n",
      "Training Observation #3830: Loss = 2.49412e-05\n",
      "Training Observation #3840: Loss = 0.0358306\n",
      "Training Observation #3850: Loss = 1.52432e-05\n",
      "Training Observation #3860: Loss = 0.00295604\n",
      "Training Observation #3870: Loss = 0.013064\n",
      "Training Observation #3880: Loss = 0.000252901\n",
      "Training Observation #3890: Loss = 0.00482161\n",
      "Training Observation #3900: Loss = 5.52446\n",
      "Training Observation #3910: Loss = 0.00075801\n",
      "Training Observation #3920: Loss = 8.04803\n",
      "Training Observation #3930: Loss = 0.000287221\n",
      "Training Observation #3940: Loss = 0.00409752\n",
      "Training Observation #3950: Loss = 0.00106913\n",
      "Training Observation #3960: Loss = 0.0029885\n",
      "Training Observation #3970: Loss = 0.0151836\n",
      "Training Observation #3980: Loss = 6.71203\n",
      "Training Observation #3990: Loss = 4.58221e-07\n",
      "Training Observation #4000: Loss = 0.00101556\n",
      "Training Observation #4010: Loss = 0.00143377\n",
      "Training Observation #4020: Loss = 0.00115651\n",
      "Training Observation #4030: Loss = 0.00648909\n",
      "Training Observation #4040: Loss = 0.242594\n",
      "Training Observation #4050: Loss = 1.93178\n",
      "Training Observation #4060: Loss = 0.000148992\n",
      "Training Observation #4070: Loss = 0.00126033\n",
      "Training Observation #4080: Loss = 4.7097\n",
      "Training Observation #4090: Loss = 0.000911132\n",
      "Training Observation #4100: Loss = 0.000194178\n",
      "Training Observation #4110: Loss = 0.232248\n",
      "Training Observation #4120: Loss = 0.00186144\n",
      "Training Observation #4130: Loss = 0.023917\n",
      "Training Observation #4140: Loss = 5.00775e-05\n",
      "Training Observation #4150: Loss = 0.000595998\n",
      "Training Observation #4160: Loss = 9.41828e-05\n",
      "Training Observation #4170: Loss = 0.0018779\n",
      "Training Observation #4180: Loss = 0.00378128\n",
      "Training Observation #4190: Loss = 0.222125\n",
      "Training Observation #4200: Loss = 0.00129744\n",
      "Training Observation #4210: Loss = 0.0139513\n",
      "Training Observation #4220: Loss = 1.85684\n",
      "Training Observation #4230: Loss = 0.000565769\n",
      "Training Observation #4240: Loss = 0.0107048\n",
      "Training Observation #4250: Loss = 0.0125024\n",
      "Training Observation #4260: Loss = 0.00474176\n",
      "Training Observation #4270: Loss = 0.00563052\n",
      "Training Observation #4280: Loss = 0.460632\n",
      "Training Observation #4290: Loss = 0.893874\n",
      "Training Observation #4300: Loss = 3.60111e-05\n",
      "Training Observation #4310: Loss = 0.037454\n",
      "Training Observation #4320: Loss = 0.252161\n",
      "Training Observation #4330: Loss = 0.0222614\n",
      "Training Observation #4340: Loss = 0.0307651\n",
      "Training Observation #4350: Loss = 0.000552436\n",
      "Training Observation #4360: Loss = 6.15293\n",
      "Training Observation #4370: Loss = 8.99724\n",
      "Training Observation #4380: Loss = 0.0107139\n",
      "Training Observation #4390: Loss = 0.00170304\n",
      "Training Observation #4400: Loss = 8.90053\n",
      "Training Observation #4410: Loss = 0.00200741\n",
      "Training Observation #4420: Loss = 2.32807\n",
      "Training Observation #4430: Loss = 4.58358\n",
      "Training Observation #4440: Loss = 0.00064684\n",
      "Training Observation #4450: Loss = 0.000307913\n"
     ]
    }
   ],
   "source": [
    "# Start Logistic Regression\n",
    "print('Starting Training Over {} Sentences.'.format(len(texts_train)))\n",
    "loss_vec = []\n",
    "train_acc_all = []\n",
    "train_acc_avg = []\n",
    "for ix, t in enumerate(vocab_processor.fit_transform(texts_train)):\n",
    "    y_data = [[target_train[ix]]]\n",
    "    \n",
    "    \n",
    "    sess.run(train_step, feed_dict={x_data: t, y_target: y_data})\n",
    "    temp_loss = sess.run(loss, feed_dict={x_data: t, y_target: y_data})\n",
    "    loss_vec.append(temp_loss)\n",
    "    \n",
    "    if (ix+1)%10==0:\n",
    "        print('Training Observation #' + str(ix+1) + ': Loss = ' + str(temp_loss))\n",
    "        \n",
    "    # Keep trailing average of past 50 observations accuracy\n",
    "    # Get prediction of single observation\n",
    "    [[temp_pred]] = sess.run(prediction, feed_dict={x_data:t, y_target:y_data})\n",
    "    # Get True/False if prediction is accurate\n",
    "    train_acc_temp = target_train[ix]==np.round(temp_pred)\n",
    "    train_acc_all.append(train_acc_temp)\n",
    "    if len(train_acc_all) >= 50:\n",
    "        train_acc_avg.append(np.mean(train_acc_all[-50:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we have a logistic model, we can evaluate the accuracy on the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting Test Set Accuracy For 1115 Sentences.\n",
      "Test Observation #50\n",
      "Test Observation #100\n",
      "Test Observation #150\n",
      "Test Observation #200\n",
      "Test Observation #250\n",
      "Test Observation #300\n",
      "Test Observation #350\n",
      "Test Observation #400\n",
      "Test Observation #450\n",
      "Test Observation #500\n",
      "Test Observation #550\n",
      "Test Observation #600\n",
      "Test Observation #650\n",
      "Test Observation #700\n",
      "Test Observation #750\n",
      "Test Observation #800\n",
      "Test Observation #850\n",
      "Test Observation #900\n",
      "Test Observation #950\n",
      "Test Observation #1000\n",
      "Test Observation #1050\n",
      "Test Observation #1100\n",
      "\n",
      "Overall Test Accuracy: 0.7910313901345292\n"
     ]
    }
   ],
   "source": [
    "# Get test set accuracy\n",
    "print('Getting Test Set Accuracy For {} Sentences.'.format(len(texts_test)))\n",
    "test_acc_all = []\n",
    "for ix, t in enumerate(vocab_processor.fit_transform(texts_test)):\n",
    "    y_data = [[target_test[ix]]]\n",
    "    \n",
    "    if (ix+1)%50==0:\n",
    "        print('Test Observation #' + str(ix+1))    \n",
    "    \n",
    "    # Keep trailing average of past 50 observations accuracy\n",
    "    # Get prediction of single observation\n",
    "    [[temp_pred]] = sess.run(prediction, feed_dict={x_data:t, y_target:y_data})\n",
    "    # Get True/False if prediction is accurate\n",
    "    test_acc_temp = target_test[ix]==np.round(temp_pred)\n",
    "    test_acc_all.append(test_acc_temp)\n",
    "\n",
    "print('\\nOverall Test Accuracy: {}'.format(np.mean(test_acc_all)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's look at the training accuracy over all the iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsfXeYFUW6/vudMwQRJAhIFFFRhFUQEF1ZRFEE8SoGVkFR\nXF1Z85qvbgAXs/f+9uqaA5hFBRNrYhV1RRcVUIIgIhiQIDkzhDnz/f7orp7qOtXxnD7nzEy9zzPP\n9OmurqrurqqvvkzMDAMDAwMDAz+kit0BAwMDA4PShyEWBgYGBgaBMMTCwMDAwCAQhlgYGBgYGATC\nEAsDAwMDg0AYYmFgYGBgEAhDLAxig4j2J6Kt+S5rYJAUiGgkEb1T7H5URxhiUUIgoo+IaAMR1Uug\n7n2JaKv0x0S0TfrdN2qdzPw9MzfMd9m4IKLb7OfqmVD9XYnoTSLaRERbiGgqER2ZRFse7X9CRDvs\n77WGiCYRUas81HmBz/UyzVh5RLqeIqL/JaL1RLSOiO70qesEIvoxbNu5gogOJCKXIxkzP83MJyXV\nZk2GIRYlAiLaD0BfAAzg1HzXz8xLmbmh+LNPd5POTdP0KZ3vfiQFIiIA5wFYD+D8BOrvBOBTAF8C\n2A9AWwD/BDCViHon0J7Xu7/E/n6dAbQA8L/5btsDXaWxcol0/lIAgwH8CkA3AGcQ0e8L0aHqND5r\nBJjZ/JXAH4DRsBajvwN4Uzp/JIBfAKSlc6cDmGsf7wHgaQAbAHwD4EYAy0K0xwAOVM49B+BBAO8C\n2AbgWFiEazaAzQCWAvirVP5Aawg5vz8B8DcA/wGwxa6nWdSy9vXf2e2tBfAnAMsAHOvzPP3tPp8H\nYA2AOsr1PwBYaLf1NSxCCQAdALxu37MWwH0e9U8AMFlz/nEAH9jH78FazOXrXwM41T7uAuB9WARt\nIYAz/d69pq1PAFwg/f4jgNn2sd93agDgBQDrAGwE8AWA5gDuBpABsAPAVgD3atoss8fKfh7v5QsA\nFyrv+ROPsicA+NE+1rYd9R0FPPcKu+9b7b8jAPwewEdSmd8AmAlgk/0sR4Ycz9p3Wux1JMm/onfA\n/NkfAlgM4DIAPQHsBrCPdG0JgAHS74kAbrKP7wLwbwBNAbQDMBe5EYsNAH4Ni+usB2sR7mr/7gZr\nQf0vu7yOAHwHoJM9maYBuC1G2UPtyXm03Yf/A1ABf2LxtD1569mTd4h0bTiAn+13SwAOAtAe1kL4\nNazd+Z6wCG8fj/rXAjhPc36A/b3qAbgQwL+la93sxaQugIYAlsPiesrsvqwDcLDXu9e05RALWFzF\nvwE8af/2+06XwyKIewBIA+gFoKFap8dzC2KxAtamZRKADtL1bQB6Sr+PArDBoy6HWOjajvOOAp7b\nNebscw6xgEUwN9njowzWRmMdgKYhxqjnO62pf0YMVQIgot/A2uG+zMyzYBGHc6QiE2ANaBBRI1hs\n/wT72lkA7mDmDcy8DMA/cuzOa8w8nZkrmXknM3/AzPPt33MAvAign8/945j5O2beDouodY9R9rcA\nXmfm/zDzTgB/8eswETUEcCaAF+zyr8Ativo9gLuYeRZbWMTMP8NadJoD+G9m3sbM5cz8qaZ+ArA3\ngJWa5lfCWmia2O0eQUTt7GvnAHiFmXcBGAJgETM/w8wV9nd+HcBQqS7Xu/d43IeIaCOs3fRSANcD\nQMB32m0/54HMnGHmmcwc1tggA+AYWKK3Q2BxYJOJKG2/lwawFlyBTQAahaxbReR3FGN8yjgFwHxm\nnmC39yyA7wGcLJXxGqO5vNNqCUMsSgMjAfyLmdfav1+wz0H6fYat+D4DwJfM/JN9rQ2sXbOAfBwH\nrvuJ6Ne24n0NEW2CtfA297n/F+l4O6zdYtSyrmdi5m2wdpReOBOWOGOK/ft5AP9FRM3s3+1hEWAV\n7WHtdDM+dVtbU2vH2VpzuTWsBXUjM2+CJao4215Ih9l9AazNQB8i2ij+AJyt1Bnm213GzE2YuS0z\nn8fM64DA7/QULNHOy0S0nIjuIqKyEG3BJq7TmHkXM28AcBUszuwg+71sB7CXdMtesLjCOIj8jmKM\nTxltAPyknPsJlj5KwGuMPoWY77S6whCLIoOI9oDFHfQjol+I6BcA1wDoRkTdAICZF8AaxCfB2q2+\nIFWxEpb4SaB9jl1SwxC/CGvH3J6ZGwN4ApYoJ0m4nomI9oQlZvPCSFiL1M/2+5sAS/Qz3L7+M4AD\nNPf9DKBDSEXp+7A4HhVnwZLRC05AcIG/gTW/Ppbammov9OKvITNfIdWVSwhoz+9kL/S3MPMhdr9O\nB3BuzDbZ/hNjYD4s8Y9AN/tc2LpkxHlHfuMz6NlWwCJQMvaFJQrz77j/O62RMMSi+DgN1s60CywW\ntzssdn8a3KKUF2ApNI+BxQ4LvAzgZiJqSkRtAcgTKx9oBGA9M+8goqNg7ZaTxkQApxHRUURUF8BY\nr4JE1AGWovMkVL2/bgD+H6re3xMAbiSiw8lCJyJqD2A6LI7hDiJqQER7EFEfj6ZugUXQx9rvuhER\nXQ2LeN8klfsnLBn3aAAv2rtvAJgMoCsRnUNEdey/3kR0cLRX4wnP70RE/YnoV0SUgqUI3g2g0r68\nCsD+XpUS0aFE1M0WOzWCpT/6CcAiu8gzAK4joja2+O0aWLvuMFDbjvOO/MbnagBMRF7P96bd3tm2\nifA5sPQcbwV1POCd1kgYYlF8jISlpFzKzL+IPwAPADhXYm0nwJLFfiCJqwBrIV0G4AdYu99JALzk\n3XFwKYA7iWgLLKukl/NYtxbMPBfWojMR1u5vnf2ne67zAMxg5qnK+7sPQE8i6szME2BZ37wEa2K/\nCkuJWQHgv2AR559h6QCGatoAMy+EZdrcC9ZiuRKWjH0AM38mldsBS85+AiQO0BZRDQQwwr73FwB3\nwlLS5gN+36kNrGfeDGvX/77Ut3sBDLfFPn/X1LuPXddmWKK8drAUyBX29Ydgif/mwzKueAPAuJB9\ndrUd8x15Pjczb7Hv/9xuo5d8IzOvgWVN9d+wxtc19rP5iTwF/N5pjQRVbXwMagKI6FIAw5g5rJKv\n5EFEe8GycOpgK6YNDAwKDMNZVHMQUWsi6mN70h4M4DoArxW7X7mCiE61RUMNYYmUvjSEwsCgeEiM\nWBDReCJaTURfe1wnIvoHES0morlE1EO6NpKIvrP/RuruN3BQF8CjsCxQPoAlBnioqD3KD06HJYJa\nBstsc7hvaQMDg0SRmBiKiI6B5TX5DDP/SnN9MIArYfkMHAnLc/ZI29xxJizZMAOYBcvpJ4wc0cDA\nwMAgASTGWTDzx7Bc9r0wBBYhYVtB2ISIWsNScL3HzOttAvEegEFJ9dPAwMDAIBjFdCJpC7eDzTL7\nnNf5LBDRKACjAGDPPffs2blz52R6amBgYFBDMWvWrLXM3CKoXLX2OGTmxwA8BgC9evXimTNnFrlH\nBgYGBtULRKR6sWtRTGuo5XB7G7ezz3mdNzAwMDAoEopJLCYDON+2ijoKwCZmXgnLwedE20u2KYAT\nURXzx8DAwMCgCEhMDEVEE2CFYWhORMsAjAFQBwCY+REAb8OyhFoMK0DX7+xr64noVgAz7KrGMrOf\notzAwMDAIGEkRiyY2dcu3o6Zc7nHtfEAxifRLwMDAwOD6DAe3AYGBgYGgTDEwsDAwMAgEIZYGBgY\nGBgEwhALA4NqgKVLl+Ltt98udjcMioCtW7figQcewI4dO4raj2rtlGdgUFvQvXt3bNiwASalQO3D\nI488ghtuuAGNGjXCyJHFi6tqOAsDg2qADRtMHM3aihUrVgAA1q5dG1AyWRhiYWBgYFDCILJSiheb\nqzTEwsCgGqGyskaneTbQIJWylmlDLAwMDEIjk8kUuwsGBYbhLAwMDCKjoqKi2F0wKDAMsTAwMIgM\nQyxqHwyxMDAwiAwjhqp9MMTCwMAgMgxnUfsgiEWxjRsMsTAwqEYothevQeEhiMXs2bOL2g9DLAwM\nqhHWrzepXWobmjdvDgAoLy8vaj8MsTAwqEYotijCoPCoW7cugCoOo1hIlFgQ0SAi+paIFhPRTZrr\nHYhoKhHNJaKPiKiddC1DRLPtv8lJ9tPAoLrAKLhrH8Q3L7aCO8m0qmkADwIYAGAZgBlENJmZF0jF\n/hfAM8z8NBH1B3AngPPsa+XM3D2p/hkYVBfIi4RRcNc+CG6y2FxlkpxFbwCLmfl7Zt4F4EUAQ5Qy\nXQB8YB9/qLluYFDrIXMThljUPggiUWzOIkli0RbAz9LvZfY5GXMAnGEfnw6gERHtbf+uT0Qziegz\nIjpN1wARjbLLzFyzZk0++25gUDKQCYQRQ9U+lIoYqtgK7usB9COirwD0A7AcgJgNHZi5F4BzANxL\nRAeoNzPzY8zci5l7tWjRomCdNjAoJAxnUbtRKmKoJJMfLQfQXvrdzj7ngJlXwOYsiKghgDOZeaN9\nbbn9/3si+gjA4QCWJNhfA4OShEwgDLGofagNYqgZADoRUUciqgtgGACXVRMRNSci0YebAYy3zzcl\nonqiDIA+AGTFuIFBrYHMWRgxVO1DjRdDMXMFgCsATAHwDYCXmXk+EY0lolPtYscC+JaIFgHYB8Dt\n9vlDAMwkojmwFN93KVZUBgXAvHnzcP3110capKtWrcKQIUPw+uuvJ9izYDz11FM488wzsW3btth1\nMDMuuugiPPXUU9rrt912G6ZNmxa7/jDtX3LJJdh7772dcz///LNn+ZUrV2L48OFYtWpVYn2q6Xjp\npZdARCVFlAVnMX369OJ2hJlrxF/Pnj3ZIL9o3rw5A+A1a9aEvuftt99mANy1a9cEexaMVCrFAHjW\nrFmx69iwYQMDYGuaZMPvWj5QXl7utCH+xo0b51l+7NixDIDvuuuuxPpU0yHe88cff1zsrjgYPXo0\nA+A2bdokUj+AmRxijS22gtughCF2NFE8R4VMfffu3Yn0KSxE33PZIRZ7d6nTT/j1ScSNMnqN3MFF\nFvnIqA06C4MagiiDVCxUxQ5NIJDLwlmKxMLveeIQdwM9im15JEP0pdibAEMsDAIRZeKIBbZUFqxc\nFny/5y7ELk/Xd7/nEX0SOZsN4qPYGwUZoi/F7pMZVQaeEItPlB1NbeEsCrHzjMpZiO9VKu++OqPY\nu3gZhrMwqDaIsqOpSZyF372F2OXp2jDEojAo9i5eRj70b/mAIRYGgaitnIUf91CIXV5UBbfRWeQP\nxV6YZYi+GM7CoORhOIv81ptL+2E4C6OzyB3FtuaTYTgLg2qD2spZ+E3OYnEWxhqqMCj2Ll6G0VlU\nI7z55psgIu0k/Pzzz3HIIYegVatWOPTQQ7Fp06Yi9DD/2L17NzZs2AAA6NevX2hP6BtuuAGA5f19\nySWXJNY/P4wePdo5fvfdd2PX4yeGkr1pk0p3uXTp0qxzU6ZM8Sy/ZIkVOs0Qi9xx9tlnY+fOnXmt\nc/LkyTjppJMiG0dMmDDBOc53n6LAEIsQ8AvpMGPGDCxcuBCrVq3C119/jalTpxawZ8lBDvm+evVq\nLF++3Kd0FTZv3uwcP/roo3nvVxjceuutzvGuXbti1yNzFuoEX7RokXM8d+7c2G34YevWrQCAkSNH\n4uOPPwYANGzY0LN848aNAQBNmzZNpD+1DYsXL85rfSNHjsS7774beXPRoEED53jjxo157VMUGGIR\nAlHEEcX2sswX1OcIuxsqtV1tvnQWaj2FEAmINq+55hr07dsXffr0CeVnUVPGYDFwwAFZmRDyBrHQ\nRx078twrpt7CEIsQ8Pu46serKRNVJQ5hBmllZWXJPX++rKHU5y+kgjudTjv/w1hDlZL3sUEVxEYq\n6tjJZDKO0YIhFiUOvwWnpnIW6qAMswAV21pDh3wpuNV6CslZhCUWYuyV4neoLkhy/gpiEXXsVFRU\noG7dugAMsSh5+MmuDWcRrUyhkZQYqhDPKr6B2FWmUqlQIUgMZxEfSb67XDgLQSyK+W0NsQgBv0xl\nxTZnSwpxOItSfBf5EkNVB87CiKFyR5KbgFw4i3r16gEwnEXJI4qis6ZwFjWFWOSLs6gOxMJwFrlD\nfnf5fo+GWPiAiAYR0bdEtJiIbtJc70BEU4loLhF9RETtpGsjieg7+29kkv0Mgh9nYcRQ0coUGvnS\nWRRTDCUTizBiqFL8DtUFfhuEXGHEUB4gojSABwGcBKALgOFE1EUp9r8AnmHmwwCMBXCnfW8zAGMA\nHAmgN4AxRFQ043HDWcTnLIr9PvIVotyPs0hqcRb1yjoLw1kkiyTNVI2C2xu9ASxm5u+ZeReAFwEM\nUcp0AfCBffyhdH0ggPeYeT0zbwDwHoBBCfZVizVr1mCvvfbCCy+84JxTPbTvuece1++33nqrIH3z\nw4ABA0BEOP3002PXMXbsWNfvMAuQLkewnwOS8Ipv0aIF3njjDc9y3bp1Q4cOHWI52E2ZMsUzb/Wc\nOXNARC4HO4EvvvgCxx13nPP7k08+cY4XLlyIv//9787vhx9+OHK/gKrn1/mm3HXXXRg1ahSAcGKo\nWbNm4dVXXwXg/lYLFixw2njooYdi9VNg48aN6NChg++3KiQWLVoEIsKcOXNc5//5z3+iQ4cOTgSC\nKKhfv75z/Ic//CHnPsoQ3/Hee+91zs2aNQtEhCZNmjjn5DGxa9cuVFZWOvfWVGLRFoA8S5fZ52TM\nAXCGfXw6gEZEtHfIe0FEo4hoJhHNlD2O84Xly5djy5YtrnPbt2/3vUd81GLi/fffBwC8/vrrseuY\nNGkSAKBz584Awg1S8Q32339/55zs0S1D5jjWrl2LZ5991rPeuXPnYunSpZ51eUHsyHVhMwDgueee\nAwDt4vfEE0+4fssLz9tvv+26JtKZRkEQ8b355pud4zBiKNmLXC7z0ksvOcdXXXVV5H7K+OGHH7B0\n6dKsjUSxIMb3888/7zp/6623YunSpU74kyjo0qVK+JHv0D1i87F+/Xrn3NNPP+3bljgviEeNFEOF\nxPUA+hHRVwD6AVgOIDTpZObHmLkXM/dq0aJF3jsnL5ADBgwAEMxCFlvskm8ceOCBAMINUlGmTZs2\nzjmv96W+Jy/P71wUjscff7xvH/yg9s+rjkMPPTRW/VGeJawYKqjuXHelpeYhLt6L+ry5LKiVlZU4\n6qijMGLEiLwvzCJUizxe5HGve6+i7H777Qeg5nIWywG0l363s885YOYVzHwGMx8O4M/2uY1h7i0E\n5MES1hqhVCZSvlBWVgYgmlNenTp1nHO5Egs/fVEQRD/iTLCwxKKsrCxW/VHuCSOGkvubyzvzQ6kl\nV/IiFrlAeEvH/a5BdQPusSSHk9c9hyhb062hZgDoREQdiagugGEAJssFiKg5EYk+3AxgvH08BcCJ\nRNTUVmyfaJ8rKHTEorZxFlFkpaKMIDB+96kTw2sBykWRLPqRD6sWr7bT6XSs+uMSC6+F0YsDS4JY\nlAqSIBZCPxD3uwbVDbi/iUws/HKu12hrKGauAHAFrEX+GwAvM/N8IhpLRKfaxY4F8C0RLQKwD4Db\n7XvXA7gVFsGZAWCsfa6gkD9ebeUsxEIVhbOQF36vCRd20OeySxbEIt+chXytEJxFGDGU3Cc/K65c\nUGqcheiH+q1yEZdVVlYWjbOoqKjImheibClYQ5UFF4kPZn4bwNvKudHS8SQAkzzuHY8qTqMoiMNZ\n1DSzxSgLruoXAOQuhpLvj/puhRgqqA+6RcVr0qqIuwON8ixhnfJ0deeTWJRacqUgziIuEa9Tp04i\nnIWOWKgbKy/T/FIgFsVWcJc0dJxF0AAqRS/mXBCHs5CJRU0UQ8l9LSsrKwkxVCF0FtWFWMR1fhN1\npdPp2N81qG61X6oYysvpV6w/NVIMVRMgfxhhfx00AGua92wcYqGy1jqEJRZJKrhFm7q2S0nBHUYM\n5RXsMp8LXqmlzA3iLOIS8WKKoQxnUU0RRwylS3u4Y8cO53wmk4lllw8A27ZtS2Sw+KVq9BJDiSxu\nMnRiqFw4i61bt7re9+7duz37KSAv8oJYCFv1yspKlJeXg5mxadMmx8kvjBhKdi6Uv186nfZ0PCwv\nL/f0y4nLWejeO+DOoCb3z8+RccuWLZHk+iK17ubNm7F169ZAn6Ok4UUsxJgJ+47ld7p27VpHwZ3v\ndLniu8i+W/L737RpU9YYVxXcRd2MMnON+OvZsyfnG1OmTGEADIDHjBnDAPjhhx92ru/evdu5Lv99\n/fXXTplJkyY555mZTznlFG7ZsmXkvmzatIkB8GmnnRZYVu3PjBkzPMu+8sorTET8zTffaOv429/+\nxgB40qRJzrX58+czAL7jjjucc4MHD3buOfnkk53jV199Vdvuww8/7OrjBRdc4Lr+0UcfMQA+7LDD\nXOV27Njh++y33nqrU/aee+5xjh988EEeMWKE9nuJbyNj77339iwjfh911FHcv39/7f2TJ092ys2Z\nMyfr+tChQ32/ka7diy++mAHw7NmzXWUXL17s2dcGDRpoz3/zzTcMgK+99lrf9+nVJ/F33333hb4/\n3zj//PMZAPfo0cN1XvTtiSeeYGbmVCrFxxxzjLaOO+64gwHw/Pnzedy4cQyAhwwZwtdccw0D4A0b\nNuStv6lUyunb2rVrmZm5fv36nmPyvvvuc8bR6NGjGQDfcMMNeeuPAICZHGKNNZyFD8SO5dJLL8XZ\nZ58NANhjjz2c62KncPLJJ7vyPv/nP/9xjuVQIYAVimD16tWR+yI8iMN4ZXfs2NH1+4MPPvAoCbz4\n4otgZm0e6aZNm+KMMywHe3n3tmLFCgDWswjIXs2yg6RsRitj7dq1WW3J+P777wFk57f22lkL/PTT\nTwCAp556Ctdff71z/t5773U8tsOgZcuWzvFhhx3meibBsbzxxhto0qSJK0SEgPCiB/Qe5MJDXkDN\nca7Loz1okBXxRrx/AfW3jN69e2vP//LLLwBy8/IHgNdeey2n+3NB27ZWUIe+ffu6zgvnN/FdKisr\nnRzmKiZPtqz5V65c6Yydu+++23Eslb2tc8U+++zjHIvxf9hhh3mWv/POOx3u6YQTTgAA7VgrFAyx\n8IFYIC+44ALsueeeAPQy9OOPPx5/+ctfnPMssZbycS6Iwn7utddeTpiOOBDPffXVV2v9LPz6QkTO\nJPYrGydQYRhkMhm0b98eI0eOBBF5LpZBkL/b0Ucf7bqWSqVw4403omXLljjggANykuGfeeaZTr9l\n7LXXXlll9913X21Zv+8RJAbMdXwWU38hnk0dO1F8g8TzV1ZWIpPJIJ1O4+CDD0br1q1D1xEWzIzG\njRu76s1kMtqNgdwnANhzzz0Dow4nDUMsfCArbMUA1IUrV3fPYUJURJ2kUQaJMP+L25Z4LiG7Vdv3\n09ukUinXAhLGiUz0WYbXIhT0LBUVFa7vIfqfy6KmTlI5sFuuJpZi56jWoeuvbgzqfoe5FjVRktd7\nlxW0hYZ4Nq9nDPNd5PdQUVHh+q5h6wiLyspKR/cg913oQ1XI1lFlZWWhw70kBUMsfCArbHWKXnlR\n1d0HeE+yODHto5QVg9KvD0FtiQEKhPcKTqfTroUuLGcRllgEvQd5wsv1RCUWcnl1kgqLGSCcNZTf\nguy1C9b118vYwK99r8VO3BN2bHg9QylwFl7PH2ahF8+fyWSQyWScd5yLQ6cX5Hkp992LWAgCBljj\nJCitbtIwxMIHcg5k3U5DF95Cvk89lpEksaisrHRxFlGhDlC1fb9JGJdYhE22FLQAyBMeiE8s5EVU\ndoZj5izOorKy0nfRDXpfujK6+uJwFvkSQ3nVU8qcRVQxVNKchUwYwnAWshiqrKzMiKFKGbIYyo+z\nUImFPAG9CEfUQRiVs8hFDKUOUCA8Z6GKoeJyFmHvU6GKofIBmVio5sFhdqBB70tXRndPPjmLqGIo\nr3pKkbOIkmRI1VmUEmdR7cRQRPQSEQ2kUvHEKSDkhcFPZ+EnhvJaZKMSi6g6C1kMFRVhOQuv3W8+\ndBZxHa1UMZToYy7DV2b/ZW4TCMcZxOEsdItCMXUWpUgsvDgL8e6j6CzEwlyqOguxBpU6Z/EkgAsB\nLCKi24jowIT7VDLQcRa5iKFyCV1RDDGUl86iOomhBHJVcKu7WJWziLNgA94Kbt2i4NWWEUPlTwyl\nchb5FkNFIRaqGKrkdRbM/C4znw0rTeovAD4koo+J6DwiSjQQYTGxa9cux+5aJhbLli1zygj7dpWz\nkAep7B0tbLrV82EQ5F8gY/369aEU3BUVFZg3bx6AKu9coMrDVBZDyV6nwnNX58HLzLGIxbx581z9\nXLdunfa++fPng5mxYcMGzJ8/33Vt9+7d+PLLL7UK7oULF2rrC4N0Oo3du3djzpw5znfzE0Nt377d\n1Z7qQ/Hjjz+66gbc7x+o8oOQIdpSIwD4ZRCUPbtliEVHZDdkZvhlm/TyBJe/9bZt2yKN0/Ly8sDs\nhxs2bPBsW9wr11FZWel47E+fPt3XT4KZnZSsmUwGGzdudBkuAPkhFsyMpUuXukROX375JZYuXYoV\nK1Z4+k5UVlY6Y0Vw+eq8YWZ8++23BSEiobYFdk6JcwCcB2AugEcBHA3g3eS6VlxcccUVTmpLWQz1\n4IMPOmWOPfZYAFULxcEHHwzA7ST173//2zkeNmyYc3zaaadF6k+/fv1Cldu9eze2bNkSKhTD5Zdf\n7ixqF154oXP+T3/6EwCgUaNGjhPiK6+84lwXDme6xaW8vNzJrgeEF0N9+eWXmDKlKmXJ6NGj1VsA\nAGeccQY+++wzNGvWDL/61a8wY8YM59rtt9+OlStXunwUunXr5tsPL4gJ3KlTJ+fbd+/eHeeccw6A\n7EVeXpSGDRuG9957z/ktp0hlZpfT5BFHHAEArvJymlDZZ0X4+nz++eeuvo4fnx2cWRDeVatWaZ9P\nDWp3zz33oGXLli5CJuOGG27Qnm/QoIFz3LhxYzRq1EhbTocuXbo4fgdeaNasGYYOHaq9Jvr6ww8/\nOOfkPOOvvvoq9t57b8+6b7nlFue4srISc+bMceaNeD9y3XExYcIEdOjQAbt373Z8Kq677jp06NAB\nmUzGc2y2a9cOt99+OwDrPevEUJMnT0bnzp3xj3/8I+d+BiGMzmIigOkAmgE4k5lPZubnmflSAN5f\noppDnoCtxMZ0AAAgAElEQVSpVAqpVMpzgRd5e4VHs7xYeuGLL77IQy+zIXZhgnAB1oTTQc7PLEPE\npznzzDPRuHFjNGnSxOU4JJLL6xz/2rdvj7POOguvvvoqAH/Ool69evjkk08c7285z7WKf/3rX86x\nvDDLnuCCG3nyySedc/fcc49nnU8//TRat27tWpAFGjVqhEaNGuGjjz5yiVpEX8W59u3bA3Bziu+8\n845nm+r7uPDCC1G/fn3X+/3oo4+c4+nTpzvH4r2r31MQx/nz52PUqFGudurXr48RI0aga9euTmpO\nIJt4vvXWWwC885WLHOn9+/d3jRvZKzmqaNWLMKmQIwXIEIRAfh/Tpk0L3b5MlCsrK9G0aVPnewoP\nbjliQ1x88sknzvHhhx/uivYAwIkOAViERWwGxAbxnHPOQePGjbWcheBAv/nmm5z7GYQwnMVjAA5h\n5luZeZl8ga10qJ4gokFE9C0RLSaimzTX9yWiD4noKyKaS0SD7fP7EVE5Ec22/x6J8lD5hryzBLIn\nmmAtxaAtZphy0Tc5PEVUB7eKigr06NHD0XscfPDBWn2LbnFo164d0uk0+vTp41lGnG/atCn69OkT\nKs+3zFn55QNo0aIF2rVr55wTu3Edzj//fPzmN7/R7oYzmQx69+6NNm3aZIkZgaoxIXbWXn1SoY6N\ndDqNjh07asdMw4YNncVLoEmTJlnvtKKiAkcffTS6dOmCDh06OP0XfWnVqhWOOOKIUCbdQejfvz9O\nPfVU53cxZei6kN9R+qOKSzOZjDN28imGkttJp9M47rjjXNdl7mfYsGHo3bs3mjZt6mzaRBQCHWch\n6i4VMdQBABxe0U51OiroJiJKA3gQwEkAugAYTkRdlGJ/gZVB73BYaVcfkq4tYebu9t8lIfqZGFQ5\npjpZkzS3iwrRtqzg9lJC+vmAyEpi1fHMj1ioVkJ+YihRJkx6zDCpWr2U237wsjBRfSl098n9kvvk\nRyy8TGL9rJ/UczrTWdEP9b1nMhmtQ1fcyAJ16tRxveMkF6mgunXEIsrcU/NfJ206C7gdfAV0xiip\nVMohFqK8TsHtlS0wCYQhFpcws6MlY+YNAC4NcV9vAIuZ+Xtm3gXgRQBDlDIMQAiYGwPwjohWRASZ\nSSZpbhcVYjDJA9Br8fIzT5UXKjWkhZ/ZpWjLy39AQPaCDkMsvHJkqFZaugXWD16267r+6fqj++ZR\nOAsgO4GSmPhexEIXklteUET/gSqipz5n3EVHZJHzqiefCOsZ7zUegqASC/k95nMuq9EA1O/qRSyE\nSFneVHn5lJQKsXA9GRGlAISxy2wL4Gfp9zL7nIxbAIwgomWw0q9eKV3raIun/k1EfXUNENEoIppJ\nRDP9LDlyhW4XKX+cUuQsZGuoqMQiF85CICiYm9jxAsGERXev13EczsKLWMTlLPzMSb04CN2ipGtX\nt2Do/APkbyQWKC+fH9mCLWjRCWsanQ8E1a0bh3E5C9XPIp9zWRVDqWNUN2bT6bRDLGQClq9Yc3EQ\nhli8R0QTiKgfEfUD8DyA94NuConhAJ5i5nYABgN41iZGKwHsa4unrgXwAhFlheFk5seYuRcz95Jl\n9PmGbhcpD6Lqyln46Sz8OAs/YiHaiiKGipKNT/RP7Yuu32GQqxgqX5yFn0hPbTesGErkIQjiLCor\nK0P7oah9SpKzCJpHOs4i7uKuiqGS4ixky0oBHbEIy1lE9ZfJBWG2YTcAuAzANfbv92CZzgZhOQBZ\nO9fOPifjIgCDAICZpxNRfQDNmXk1gJ32+VlEtATAQQBmhmg3L5Bfvk5nIX80dYAVk7NQw1EA0YMZ\nqiEIonAWSYmhvPqdK2eRqxgqXzqLKGIoHWehE0PJDoQqUQwbvkVFIYlFWDFUWM6isrIyS/QkH+uc\n8vI9l2WfLYFcOIuSIhbMnAFwv/0XBTMAdCKijrCIxDBYvhoylgI4HsBTRHQIgPoA1hBRCwDrmTlD\nRPsD6ATg+4jt5w26XaQ8uWXKT0RF5SzE4PaaFDKS1FnEEUPF4Sxy1VnkKobKB2fh9X69xFA6nYVO\nDCWHJvFTcMucRdCioz5bqXEWfv1RiYX8rKoYqpCchddmRMdZeEU6KAkxFBEdQEQv2qati8Rf0H3M\nXAHgCgBTAHwDy+ppPhGNJSJhe3cdgIuJaA6ACQAusNP8HQNgLhHNBjAJlpI9fymrNFixYgVmzZoF\nwMrOJn8UdRe5evVqJ5ObfB6wPtqCBQuS7KpvZjTdQuOVmc9rYi1YsCArJ8RXX33l/I4ihpI93mUs\nWbIka+ceRhcCuB2v5LIrV66MRSxkb+lVq1bhxx9/xIIFCyLpLL7++mvnmo5YXHrppZgyZYrWGzmd\nTrsyuYn+6NplZnzwwQeoqKjAhx9+iD//+c+YN2+eVgwlPL2DxFCbNm3yJRY6LltA+DXIZb799tus\nOuIgrM5Cnovfffdd6PrkebFgwQIsXbo0i7Pw8/0JiyCdhe47b9myBZ999pmrLzouWEQxkJ1/k0IY\nncVTsOJDESwz2JcB6L25FDDz28x8EDMfwMy32+dGM/Nk+3gBM/dh5m62iey/7POvMHNX+1wPZtZ7\n5eQRAwcORK9evVBZWel4/QoIkUzz5s0BAPfddx+uvLJKF6867sgpNZOAzolMQMdZyFn8/PDqq69i\n48aN2Lp1q8vx7Y033gAATJw40dWGbjKLbG6ifXkRlfHdd985DnWqzmLChAm+/dQRLsCaYGq6VkDv\na/HrX/8aQFU4jJUrVwIAWrVq5XhYiwVQfHcZwjZe/L/qqquca7rJ/8gjj2DQoEEuhzbhJCfS3n74\n4YcAqsKDDB8+PKuen376CcuXL8f999+P/v3744477gAAZ2ERYVlWr16N+++3hAHTpk3zFUNdd911\nzrGOWMibnx49eriu/fjjj5g9ezZmz57tnOvZs2dWHSpEmA3A28s8KP2w/AzCIVOE6JEd3QTU8SqP\nC7EBEeNHePDHSYGsQo4okEqlHOdKAd14keefGH/qN9y+fTsefvhhAHBCnCSJMMSiATNPAQBmXsLM\nf4FFNGoUxKKmWwAFsTj99NMBWIN0586d2GeffbJiu5x++umu3yeffDJat26NDRs2YMaMGbG5jq5d\nu4YqJ3MWUb06Fy5ciPLycgD6ySZ2jDoxVKdOndC5c2c88MADAKzd1MEHH+zpFFe/fn0cf/zxALLF\nUFHekdyHPfbYQ+s9/+WXX2adEwT9xBNPBKCPc/XXv/4VADB06NCsdzlw4EAAei92eTFQPcjFYv7O\nO+9kxasSoSVEnbfddltW3QJqXCwxPsTzZzIZJ0rAihUrfDmLzz//3Fd0JscFE+FJ5HNr1qxxxbZS\n41zpIHPHXuXD+lkA2d9vwoQJnuNLQBdCR7z7OnXqgIhc4UziQh4PgwcPRrNmzVy55dPpNNatW6eN\n47Vo0SLHiU/9hnLUgKlTp+bczyCEIRY7bQulJUR0CRGdAiB8AJhqBj/Wl4iw3377OcrDHj16OLl6\nBVq1apW1O2vdujWaNGmCXr164ZBDDonVr7DWKrKcWg7FEAbM7Dy/bpKIPug4C2ZGjx49XCa7zZo1\n8zX1EyEuouosZKjKTV3uanUnB1Q9n5pfQFeGiNC5c2fXt/NLqiR/fzW/shBD7bPPPr7RRrt27epr\ngquOMVGXsIKTdRZE5MtZBL133btp2LBhVp+jQDVbDduuV5s6/4ODDjoIABxOUWdFps5feaO31157\n5UXBrTOGEZ72gPUumjVrpo2T1alTJ5doV35meQyIZ00SYUxHrgGwJ4CrANwOy4nuQt87qjGCFFpC\nGemlTNUpK6PK0XWI6oMglJpRUFlZ6atcVa1P1IVabc8vWYv8/lRiEUVZF8Z01u/9+5nt5uO7qXWo\nSksddO9ShdfiLCtmwyq4M5mMr84ijJI36qKqhmnRIQpnoatDtSrTmZ2qDnF+hh1x4Wdmrx77we8b\nFiJUvC+xsEN2nM7MnwPYAivqbI1G0OAQZqReGdlUM1PVAiMuwk5GebGP0668wHhd04mhZOshAS9L\nI8Df6zhOf8X9XmaIXvDjauJm3NM5bAoI0YGubjkfdNAC4rWQyiafKrEI42ehIxZhvksSnIXaR3VM\nqhsFL3htCHTEws8ZNS7kdlUFuty/IOhywUetIxf4ria22exxfmVqGoIGh9ht+C1MqpNQPohFVB8E\nXViBIMhiKF2fVY5CXXx0xCKMl7g6maMkKgrDWfi9fz9CpX7fqKJAIB5nEYYb9eK+ZM5CzhKYTqcd\nJz21j6o4UUWUjHNhIT9fmORFujLye/Kbt15ldHNYXcTzzVnoiEXY9UGdT3Lfis5Z2JhFRK8CmAjA\n0UQJi6aahiichW5C6ziLQoqh1N1kFDCzVgwlOIRCiaGiIIxTXqHFULrFQUB1tJIhcxZxxVB+nIW4\nT+X4wpqoxumPF8KEDAlyHBScgZiPXogrhsoXZ6HjAOTvmw/OolSIRSNYRGKwdI4B1EhiISI9eqFY\nnEUcMVQczkInhhKDVPQhH2Io+f15EQu/+wXCOOUVWgzlJx7w4yxkYhxXDOWls5CJok7ZXWidhVdg\nSK86vTiLunXrYseOHb7te33jTCbjK4bKF2ehE0OpvhdhUPKcBTPXeD2FjKB0p2F1FiI4W2VlZexF\nR0YhFNxeYqh02korGsRZhCUWQpHupbMQC5YfZyK3Kx97xdnxQhQxVFjoFgcBP52FQL50FrIYSl0w\nvRTcunqDvoG8yQiLqDoLXZlMJjuntQ5+nIX6npPQWfhxmkD4hT6VSnnGwcolx3xYhPHgfkz3l3jP\nioSg9ITpdBoLFizATz/95LuLlSdl2MEwadIk3HXXXa4UpgJ+k3HBggWOH4DIKhZHwf3AAw84/iby\ns4l6/v3vf+P555937PflPNm659SFJwCqBrlODLVt2zY8/fTTAMIt1kuWLMELL7yAGTNmYOnSpTlZ\nQ6m76rhiKLketY4333zTs+5nnnnG6Uuu1lALFizQiqGELb98//r1652UtkOHDsXYsWNdderygYft\njxeEPw/g9tnwqlP2xSgvL8fw4cPx008/OcRi9erVgXqcK664wuXDo3vP6riXvcMBK/3t5s2b8emn\nn2LatGmOI6UOzz77LJ555hnHcVKt3++cDqlUyuWbM3lyYYU7YVaTqdLfpwBawg7yVxMRRCx++eUX\nZ6DrvDvFZBQLqZ8YSngNC/z2t7/FzTffjKFDh+Lnn392XVN3OLITUteuXZ3UrmKib9++PfJuY/36\n9Y4zntznyy+/HIDlvDVixAhtv6JwFkuWLAHgXgAEYRk1apTzXsQkE+0Lr2sZ48aNw7nnnutkExML\nrgxhx37VVVdh+PDhrpSzMqFS03eqfhA33ZSV7NEFsajKjohqpjsB2aZe5PX+9NNPUV5e7stZ9O3b\n19VvgZEjRwKA4zcwbdo0HHrooQCAiy66yGlPECuvxb28vBxjxoxxecKL7yFnIJQRh7OQx7e86ZAh\njx0RigcATjrpJLz44osAgJYtWwIA5s2bh8WLF7vu/+1vfwug6p29+eabLudW3dyU/R9WrVrlmkMr\nV67EiSeeiHPOOQe/+c1vcMwxx6B///4uwifj/PPPd74LYPkdye3tt99+SKfTWb44ADBkyBDHqVBA\nXnsA4Prrr9e2mxQCiQUzvyT9PQ3gDADB/vw1FP3793cCBqphQYAqD1Ch+/BTcOu8hgVUcZi66Opi\nDMkQ7Oo555zjOCXJELuwP//5z9r75UF90UUXebYTh1iIvvfq1cvVXmVlpWvndPXVV4OZHa/w//zn\nPxgyxMqf9cILL6BFixZZE7VVq1ZZ7RERfvnlF9x77714/vnnXSFIZDGU7FU8YcKErOc577zzsGPH\njiy91mOPWYy2+GZ16tTBueeeC2ZGt27dUF5e7gpRctppp7mcHuVJL3RDXmNGcJ3q4nzGGWcAsHJH\nd+rUCWVlZQ7hOPvss51UqKre6bXXXtO2I+JKAVU7XxFKQ4Wszxo0aFAojjCM6aiXT4Hs/Sw2Sel0\n2unzpEmTAFhzlZkdoqGrX2777rvvxqBBg5zf3bp1c/VTjDU1wkCQnhOwnOZWrlzpIj6LFi3Ctm3b\ntGl9X331VYfbE+jXr19eRNpxEafljgCiuQbXINSpU8cRWfiZacryfS/Owm83FhTdM0jxJohBWVmZ\nlj0X9Xl5EevEUDrIi09Y01lVDCXaCCN+Ec+yxx57aDOMeXFTclndQlVZWema9Gq8LwHd+xKiEK9v\nXr9+fVd9aggU+T2IXbrXAir67rdANWjQwKXglkWSqs5C9lj2ghhrfuNY1LfHHnuE4jKCHOoAbwW3\nfF7WWYgyunHoVb/8TOqiXbduXRf366XXCfO8devWdUU3APQZ8gR079prLhcKgcSCiDbAsn4CLE5k\nPQB/frwGo6yszNf8UaezCLNzCoLORtwPspLYb8H2WgCCTPvq1q2LXbt2+RJFLwW1zjxXlA0SnemU\ntjKi6hlkziJIERlUh5+eSv6t1i1fq6ys9N1giPN+xEIoZnXGDmofvZ5TNW0NUsiLesvKypzNlN+3\nDFJe+5WRCYccrkWUCRPVVdTvtylSFdxRM07KyIcC2s9vqRAIMyPkkJuVXEzSVgKQ2V0/ywZ5xx2H\nswgiDmE5C68FW7e7lxHEWQhiEUcM5WVxFWUieBGLqEp9L84iDrEI4rJ0x+pvQSyCvoufGFIkU9IR\nC3VchtnIeFn+CcjEQuyWg/yLgsxidX3QHYv2ZM7Cjxir9ctl1QXdy3RWXQLDWEzlY9kMYx2YJMLM\nrJMBNGTmDDMzETUhov9KumOlirKyMkc2HcYaym+X6DeAdKxuGHNDtW6vRVi3u5eh+lmoEOKYfIuh\ncuUsohILecetS2gVBmG+uers5XVNLLxBxMKPsxCLnBgzwoNb7mMQZ6EuzkEe5+KbysTCD2E4izBi\nKOFPJDvmxRVDqfNR5SxEWbWclw9IvhHG7yhJhJlZY5nZCZbOzBsB3JpclwqPKFQ/nU679AEqdOx+\nHDGUzoFIridfnEVcMZS8KIgwErmKoaISC13ZqOy+vONOkrPwe59JiqFEeS8xlN9CKh9H5SyiOlMG\nlVH7IyCeQeak4oqhdKbTugyIYTiLpIhFMcVQYYiFbvaFmklENIiIviWixUSUpecgon2J6EMi+srO\nxDdYunazfd+3RDQwTHtxEeUDBFlxqAtHXAW3Tgwlt50vnUVcMZTs6ORVVxQxVFgWO9+chbzjjstZ\nhNFZROEschVDyYunzpdFtJMvziIOsfBa/L3KeBEU8Qx+nIXXmFDnptoPlbMQYy8MZ5EEByDPkWJo\nA8LMrK+I6B4i6mD//Q+Ar4JusiPWPggrUVIXAMOJqItS7C+w0q0eDitH90P2vV3s310BDALwkF1f\nIoji0u836eXr+eYs1HrCchb5EEPpyuhiEOUihhJlw1pDJS2GisJZRBVD+YlJ8iGGEjoLuR4vjter\nHZVY+L0P2Roqjhgqqs5ChngGP51FWM5C7bPKWYjrxSIWYo7IkRYKiTAz4goAtwB4A5ZV1HsALgtx\nX28Ai5n5ewAgohcBDAEgGykzrPwYANAYgDB0HwLgRWbeCeAHIlps1zc9RLuRERTiQ4YaDkBFmF2m\nwNSpU9G9e3dkMpksZ0B54H7//fdZprorVqzAnnvu6XIikuElhnrppZdw1FFHucwh69Wrl/UOgnQW\n4tllebVODKXLF/7oo48CcE/itWvXunJre0FOOqTLtxzVDj3fYqgVK1Zg/fr1WeIw+d2oXIF87f77\n78fChQvRvXt3bVuiXjmNqa4/n332GT777DPHNDaqNZRwyluzZg0effRRTxNrAJg+fbrjEyCIxZo1\na7T+AwLy+P7xxx+1ZWR/hssvvxxz5szJyhgpnuutt95yrkUhFkE6i8WLF2PMmDGoW7eu49iojumV\nK1dmZWhMSgwl+lkMcVQYp7ytzHy9nQ/7cGa+kZm3hqi7LQDZDXmZfU7GLQBGENEyAG8DEImtw9wL\nIhpFRDOJaOaaNWtCdEkP1aVfxmWXuemi7KykYwXDiKHOPPNMAFXOWE8++SSuvfZaVxl5gX/uuecA\nAGPGjHHOfffdd1mEQk4uf8IJJwDI3t0PGzYMvXv3dhzT6tev7yzeMoLEUPLziYVl8+bNrjK6fNjM\n7OShjsoFANY7O/DAA7NyQcvXo0DmCmQnS5FLPGodv//97wEAH3/8sauM7LH97rvvuq6JPN5AVRpW\nr7zUOgwdOtT1+/PPP3eOxXj1IhbCA1qF8NoWY0O3oRKe9Q899JDjlS+eZfp0/31dGDNlNezIY489\nhj/+8Y+uPqdSKWzZsgXLly938lDL71OU8eqDfE1EARAQZrljx47FX/7yF1x55ZXQIUy+b78UuWHh\nZeZ95JFH5lx3qPaDChDRu0TURPrdlIjeylP7wwE8xcztYEW1fdZO4RoKzPwYM/di5l4tWrSI3Qkd\nS/fMM8+AmfHggw+6zh9++OHOsS5FahgxlBwrBsheZOX7gSo295prrnF2NTpFrij3wAMPOKEZdHLO\n1atXO/X36tULI0eOxDvvvOOqK0gMJUJJyGKoTp06ucr86le/yrrPL25SGBx33HH47rvvPFPG9u/f\nP1J98iIqUrLOnDkzUkpaeRKL3bA8TgBr4XnrLWvayOFGAMuRTQ1ToosOIHDNNdc4x8yMiRMnuq4f\nddRRzrH4Jl6ms82bV1nGjx49Oqst3dgUUPOLA1WhNYJk6rnsjMXYA6znuvrqq10Oa+q3k8eZvAkQ\nc1MYaMjvDbA2VmGg26jKa0qbNm2cyAO5QKcnBKpCuCSNMLz2PrYFFACAmTcQUZsQ9y0HIAfGaWef\nk3ERLJ0EmHk6EdWH5dcR5t68QTdw41oJeSkSZYSx2JH7pJM9R1Ec62StqnzXz4ksSMEti7x0dcjs\nvmr2mG/E1VnoLIei1iHrPXTPJr6D7vur5f3EYHGU716ms3JdsqgpjB5P18eoOos6depEDgMui/HS\n6bSjowljOqsq1v2+ddhx4DcXo9QTBC9ikcQ80iHMU1QSkRNBjIjC8uczAHQioo5EVBeWwloNk7gU\nwPF2vYcAqA9gjV1uGBHVI6KOADoB+CJku5Gh+9hxrYTCmFH6ybN1fZIHtTxgvO7RKY5FXwTUiRXF\niUw+J+ssvOrwM3vMN3KxhgpS+oepw8t8U1z36qOfQlxFFH2KrOOR+yCIhdyuHI4ijALVz6Q6rDVU\n3bp1IytrZWIhfEj8TGe9LJ6CHAfDjgO/cDpR6gmCvCmR609iHukQZtSNBvApEX0Ay4z2WIRQcDNz\nBRFdAWAKgDSA8cw8n4jGAphpZ9q7DsDjRHQNLGX3BbaH+HwiehmWMrwCwOV2itdEEIWziLKIiv9B\nH1O301Q5C68dou4elaDpzO3UieVnbuin4NYRMgFdX5OYRDJy8bPIlbOQleR+nIUfpyaQC2chf2fV\nL8VvXMrEQmwm/MRJuueIylnUrVs3MmchGyIwc6DprJfFU9DczGV81kTOIkzyo7eIqDcAER/6RmbO\njs2tv/dtWIpr+dxo6XgBgD4e994O4PYw7eSKuGKoMJyFjtUNs5Py2gFFFUMFcRZxxVC6JDtedXhx\nFqUkhso3Z5EksYgbeVQWSeoWyqhiKCJyREBq34KIhcxZ5CKGqqysdBELnUl10mIoHZIY58UmFqHe\nBjOvYubXAcwGcBERzUm2W4VFPsVQYRTc6kQK4ixyEUOlUilHgZdPMZTOdDaqGKoUiIVODJULwfEy\nIwbgSVTlfnj9lhFELOTxJB8HEQudGCqIU/MicmE9uHMVQ8km5Tt37gzMwZ6EGEqHJMREurEqn08a\nYayh9iGiK4loOoCFABoAuCDpjhUS+RRDhVFwq6x9GDGUuqhHEUOJayoBAuKLoWRC4OeUp3sWv3pz\nRS4iJL+FPmwdfrtkPwW32ma+FNxqG1HFUEFQ+xlFDCUU7LlwFkIMBVjEwm9Oqv0K4izyJYbKV8pT\nL9PZQuksPFshoguJ6D0A/4Hl43A5gJXM/FdmDvTgrk6IwllEFUPF5SzUHZAqT48ihhJ1RLGGkvus\n65+8g/RzylP7Wh04i3yIoXSIwlkkJYbyE4/KYqiwu30vYhFGwZ1KVYXqiAKdGArIP2eRLzFUvlBs\nMZTfqHsUFqEYKogDEdXI8OTjxo3LOpergnvHjh0YMGAAtmzZEpjIKGkxFGBllpPtsa+44gpXWT/O\nQgdR/ogjjsBdd92lraMYYqi4Cm7hTCefi1pHJpPxjdkURWeRi3jEK6mOrL8Sjp4yZM7iiSeewHXX\nXefbDgDHEU5tO4izuPPOOwFY7+L1118PbEeG7EEtE/lnn3028N3K42/FihV55Sx27dqF5s2bY9Kk\nSVi+vMrS/9tvv41UT1B/Pv/8c5cvSb44lyD4zYq2ACYCeICIFhDRGADeqZ2qMUQ+3zZt2qBz5844\n4ogjtA5lQHjT2WeffRbvv/8+AOCHH35wlRGpV5s0sXwdkxRDiUxfF1xwgZNuUobwLFYnRtu2WQ7z\nAKoWFNnJSOSmjiqGatMmjLuON3TZ7NQsdEHQOXPG5Sxkr2vdQjt48GD06dMHt956a9Y11WO8YcOG\nnu0FETM5Va5KBHXcz2WXXYa+ffvi6KOPdpwav/nmG1eZhx9+2LdNgbAKboEdO3YEhvjWhQ1p27Yt\nevfujSFDhmDmzJkArDTFW7dmB5coKyvD0KFDccghh2T1SxdlQCDsOBDveMKECdiyZQsGDhzoyhme\nLxxxxBEALO/4pUuXAkBW9r0k4TnqmHk1Mz/AzH1gBQPcAWAdEc2zzV9rHF577TV88803+OKLLzzD\nPYQVQ8k7TNWzuW7dujj66KPRs6d3KnMvMZQgLJlMBi1atHCFKNCJg8IqKNVnkXNEA3CU5Dt37gQz\nZ8XC0dXhJ4Z6/PHHXSk9R4wY4dtPHZ588kkAwMknn+wZJj0IusmWi4JbQDd+9tprL3zyySdZ4wEA\n2vw3CD8AACAASURBVLdv7/rtFRsKqIoccMwxx2ivH3nkkc77kImWIBbqgvnggw/i448/RvPmzTF1\n6tSsxXnkyJG45JJLPPsjI6yCW+Css87SLsry/Tov8nvuuQeff/45jj76aFfYG7ERk0FEmDhxIk4/\n/fQsXYLI4a1D0DgQ4U7Et9YFHcwnDjvsMOy5554ugi/nk08aYa2hfmLmu5m5O4DsQCg1BGF2EmHF\nUEHhDtLptKesH/BXxIl7Mxl92PIgfYNXf6IgiihFRyyC9DhhkFTy+rjEIpew0eqzJGk6G/Suo/TF\n696w31MWjckIUnp79SnIMED2OVKDc+rK+0HdDHk5/+UT4n3F1a/lgsijzvaNyA4iUwMQZpEIK4YK\nU4+feaKXGEq+V7Vo0S3EYRewYhOLOAttUsQiFwV3vtpMwqTTSwyVS190bQD+i6VqERcULiOoj2HN\nSHXGHn7zNcxGi4i08zgJzgKoel9xLfdyartgLVUDhJkUOksjr+tB9fgNKC8xlHyvel6nQA27CEcd\ndGHMP3Ximbi+DDokNVFyUXDHRRRrqLgKzUJwFmEIp7q4695bFM5Crs+vrzp/mFx25kLsqRvTSREL\nWaogfhcKhlhIyAdnEUcMFcUaSr5XPV9IzkI3GQrNWSSFuJxFVH8BGX6my16I+s7ErjRoIcsHZxGF\nWAiRkIwkOQs/3yAZYYiylzi5VoqhiOgwzelNAH5m5sJn4EgQpSSG8rMHl8VQ8r2FJBa6eguts0jK\nZDAuZ5ELsVCRhIhNLDRBC1kunEUYLkvnUKaO8SichTwWwzgz+oWoiQpVDyJQE8VQYUbBOADdAcyH\nFUjwEFgB/hoR0Shmnppg/wqKfCi4wyo7ZcVeHM5C7C50clKvXZcfog66KJyFTgyVjx1RUtxIXGLh\nl+o0KsIs0HF8SpIWQ6kBC3XQcQKZTCY2sQgrhsqHA6YMoSCXveJ1fconSl0M9SOAnnamvG4AegJY\nBGAggP+XYN8KjjCLRNgwGGHqiavglncXOtY3js4i6sITJkSKn+msWra2i6FUJBUKJWkFt2gnLGfh\npeNIQgwlj8d8bVrkZ9VFdc43iimGCjMiD2HmueIHM88D0IWZFyfXrcJBTkWaT85i8uSq1B26/MWZ\nTAazZ8/GG2+8oV0o5861Xvm6deswceJEVz8By75aJSJyKlUBL29eFfkQ6XjJ3R955BHnnPAcjyOj\nVyHu8cv1HAdxOQvZGS5JiPaaNWsW+b4wnEWuxELebc+fP9/hNtScGqJPAPDyyy+jXr16ICL8/e9/\nx6effuqqUx2fXvMwjM7ilVdeQdOmTV3t6xB2Tnz55ZdIp9P43e9+55yTPbjziaVLl+Kpp54qWWuo\nhUR0PxH1sf/+YZ+rByvXRLWGGpUzCPvtt59veeGVLePiiy/OOvfzzz8DsByehGNYmzZtHG9Qscjf\nf//9AIBffvnFuXf9+vVo3rw5KisrHa9OGXLayQsvvDDwmUTbgwcPDlUWAA488MAshyaRllRApKkU\nqUYBOIvA/vvv7yp73333oV69ejj++OOx//77h/KA7du3L0aMGKFNBxoFkydPdjnB6Yi7H1Rvd933\nDgORY71x48aeHvSAlSv6d7/7nRMyIyxUYuHl1CePH8Dfm1x4dqdSKec7yBZCJ510UtY94lrbtm2d\nBXzkyJGOI+t1112H7777DgDwt7/9DYBFTGTnOzntrJyru2vXrp59FfnIL7usKh2PH3E54IADcOON\nN2LgwIHa68yMzZs3Y+fOnVkEeNq0aY6DoAiHk0+UqhjqfADLANxk/60AMBIWoTg+ua4VBk2aNHFC\nXoQhFrLHr668ziNY9c4FgKOPPhqAFVtHDLRPPvnE8Wz22zkcdthhjny8Tp06ePnll13X5UV7//33\nD5WXOpVK4ZVXXgksJ0BEmD9/Pv74xz8659RFpXXr1jj22GO1O1l1MWzZsiV27NiB999/H0uWLEGP\nHj0C+9CoUSM8++yz2lzoUXDKKafgq6+qYmNG5bJUOfkFF1wQqx+33HILmBkbN270XQTq1auH8ePH\na72V/aCKoc477zxtuUsvvdT1W80nLuOSSy4BMyOTyTgLuyya0X17ce2mm27yfE5R5q9//SsAYOjQ\noRg/frxzXcxZwM1hDR8+3LOvgtjs3LnTORdkDXX33XfjgAMOcJ1/4IEHnOODDjrIU/xYp04dMDP+\n+7//27ONuChJayhm3g7gbvtPxSbNOQdENAjAfQDSAJ5g5ruU6/8H4Dj7ZwMALZm5iX0tA2CefW0p\nM58a1NdcEVdWLSMOW6haZgRZSqXTaWcXlkqlAvsd9rlyHXhe70On+E3Koa4UUMgJHAWq6axXP/Mp\nhvKL+ySXU1FRUYF0Ou0a/1HFxCriWi966dOEn4WXYUOSIqKStIYioqMAjAHQQS7PzAcF3JcG8CCA\nAbA4kxlENNn2ABd1XCOVvxKAvIUpt8OLJA4xIOPKqoPOBUF16pEnka6+srKykiQWXuGhw1hO1SSU\nKiFUlaNeY9Uvh3UYyJyFTtErt+/HWcTpR9Q8IHH8WcR6IYiFF2dRCGJRUpwFgCcB3AhgFoAoKv7e\nABYz8/cAQEQvAhgCy+xWh+GwiFLRUCrEQp5sXhZXglik0+nANqN4lecCL85Ct2CU6oKaD5QqIVTF\nUF7fW+1/XBNdQM9ZyO17vSvBWaj1BiEqZ5Gr8yMReYalT3IclKrOYjMz/5OZV9jpVVcx86rg29AW\nwM/S72X2uSwQUQcAHQF8IJ2uT0QziegzIjrN475RdpmZa9asCdElf+RDDBV2YukceHRiqEJyFrnC\ncBYWSpUQqgpur2+Qa//lb64T0cjtexGAJDiLfIuhxP3F4Cz8fLSSQphR8QER3QngVQCOZkg2p80D\nhgGYxMzyFrQDMy8nov3tPsxj5iXyTcz8GIDHAKBXr16xDfXFC4+bOCfoXBDUDx/ksKdyFqVCLLw4\ni9pGLEr12VRiEZaziNOO2Oz46Syicha56iwKLYZKmrNIpVIlRyx+o/wHAAagt7urwnIAshlQO/uc\nDsNgpW2taoB5uf3/eyL6CJY+Y0n2rcVDLsRC/siqgjtIDFVWVubs2FKpVN7EULlCt6vzEkMVcpAX\nAmVlZVmpaksNYcVQ+eQs/CLK+o3LQnEW+SAWxVJwF3pTEsYaqm/MumcA6EREHWERiWEAzlELEVFn\nAE0BTJfONQWwnZl3ElFzAH0A3BOzH4GIu3CFibwaBn5iKC/OQpj/FVsMJU8kr8mYVOiDUkKdOnUc\nYlFdOIuw1lBxAhaGtYbymnulpLPwG7+1iVh4Pg0RDbf/X6X7C6qYmSsAXAFgCoBvALzMzPOJaCwR\nyWawwwC8yO4ReQiAmUQ0B8CHAO6SrajyjebNmwPIz8cNS3jkdJ5XXnmlq/3ly5dj/PjxWLJkCa6/\n/vqse1OpFJYssZisJMRQUVI1yhn1vIhnUqEPSgnyjraUOYtMJoMBAwY4v3VQx0vU1J3Lly/HuHHj\nsGDBAm3qW5mz8Jovjz/+uMsR1a+/MqJaQ4UhhKqzqRjzjRo1AhFh/fr12vumTZsWWHdc/M///I/j\nZFgo+I3qpvb/7CTFIcHMbwN4Wzk3Wvl9i+a+/wA4VD2fFN577z288847TgiAIDz77LOeA0TFF198\noT1/9dVX47bbbkPXrl0xf/58AMhqX070fsMNNzjHatROeRLpHK2iEMEnnngCBx3kaxXtwqWXXoq3\n3nrLlZNbRnXiLN58883QY0BFx44dMXfuXDRt2lTrhFkKEN9C5Ar32kTIG5kxY8Y4nvhRMWfOHPzp\nT39yHOsEwnAWOvhteiZOnIivv/7aN6+7HCZFzN8w4XCuvfZaVFRU4JVXXsFzzz2Hfv36YcWKFbjq\nqqtw3HHHBd6fT4wYMQLPPfdcQdsU8CQWzPyQ/f+vXmVqCvbdd1/84Q9/CF0+KF90vXr1sHPnTixb\ntswzbMPee++N7t27uyaLOnHkRVYOxdG9e3f861//ApDNWYwaNSqrLXWSffTRRzj22GO1/brooos8\nnkqPfffd14ljpYOq4N53330LPsHC4uSTT459b9++fTF37lz87W9/K1l9jPotvDYR8gJ6yy23xG6P\niJzQKVdddZUTGkQWuwquPgxkMa2KoUOHYujQob73i/vkzVaYzUH79u1x//33O6F3AMv7XO5ToXDt\ntdeWHrEQsHUGFwLYD26nvOxVyQBAFWsbJI7ws6QA9HH/1XpVnYWX+arabqGgiqHU4Ic1DaVKKIDs\n8ZaUgltATjlaVlaWpfROpVKxcmXkkikQcOtRcv1ehSYWhW5PRpgv9QaAzwB8gmhOebUWYlIETYR0\nOu0rd5QXWa+wB6oYKkwIkkIOOFUMpYZVr2kopVDrKlKplMuBLKyCOy7k0CJ16tTJihcVRt8mI1/E\nQlZIVzdiUcyNVhhisSczX5d4T2oQwgb5ClL+eokMZCKkTrhS5yyKYcVRCJQyRyGgWu4k5WchIFte\nCdEWM7vEUHHye8cdv+J+L4/rOCj0dy/m3Anz1t8hohMT70kNQljOIohYhOUs5N9hHI8KTSxUzqIm\nEovqAJWz8BoH+VoAZc5CzAXVz6MYnIWM6sZZFJMrD9PyJQDeJaKtRLSeiDYQUThToFqOMGKoOMRC\n5Sy8uA6BUhJDVVZW1mgxVCkjrBgqX9ARCzXqbRydRdzxUxOIRamLocKbKxi4EEYM5afg9jI5rW6c\nRW0QQ1UHyGFigOTHgeAiiMj55mq4kUJyFrnEcgvqU6FQksSCiDox83cAvFJP5TM2VI1EGGc5L+9P\nQB+EDahe1lDpdBoVFRVgZsc6xhCL4iCVSrkS/yQ9DgQXIXO/qhgqjs4in2KoXFGbxFB+X+omABfB\nykmhIkxsqFqLAw44AEuWLAkVs0n1UpUhp0SVs9CVl5c7x3Xr1nV52Oq8bVWLqz333NO3X/nE7t27\nsW7dOgwYMABTpkzBtm3baqRHd+vWrQFY/jOlikwm48oNHbToRk0vq2tPiB23b98OABg3bhxWrFgB\nwBqrUdoQSvLevXvH6o9uPuaav/29997L6f6oKEnOgpkvsv/HjQ1VazFt2jRfRzUBefAefPDBzvHL\nL7+Ms846y1VWzotcv3595/iEE05wLf7t2rXLaqdjx46u3wcddBBmzpwZOYxDHGzbtg0AMHXqVNeu\ntqbhxhtvRIcOHXzTehYbat7zTp06eZb94IMPssqHxRdffIHevXs7IqdUKuUQqTvvvNOJMnDUUUe5\nHAAvvvhiLFu2DO+88w4AYMaMGa56W7ZsiaeeesrToTQI8kJ75JFH4oYbbsBhhx0Wq64wePzxx/Ne\np/wMuvzmSSIUD2gH++sCwFmlmPmFpDpV3dG6dWtnp+kH+cOfempVuKw+ffq4yh1zjJuJEzmHzzvv\nvKy817rdoi78RM+ePQP7lw/IfgeCo9ARtOqOsrIynHvuucXuhi+6d3cnnvTb1efiZS9yVmcyGcev\nRhVDNWjQIGuzcvfdd+MCO3/5+PHj0atXr6y6R44cGbtf8uasd+/eOPPMM2PXpeKiiy7CuHHjAFjj\ne9myZejXr1/e6heQn+GEE07Ie/1+COPB/RcAJwLoDCso4EBYDnqGWOQI+cP7Odap7HNQPgIVxYzN\npEvyZHQWxYGqH0gq4KEcVkPoLOSw3l5e/EGGGvnqF5D/Z5frDms6n2s7hQ5YGWa1ORvAcQBWMvN5\nALoBKJzQuwZD/vB+iem9iEXYCVUqxKIYSeYNqlBoYiGLoWRi4eXFnzSxCDIxz1fdtZlYlNsZ7CqI\nqBGAXwB0SLZbtQNenEWQX0TUlIrFJBaqQx5gOItiIU7muTgQ41VWcKtiqKCwNElbLuX72eWFO0kO\nOslnCEIY0vQVETUBMB7ATACbAejjbhtEQlgxlEoU1Mx6QShmvCIjhiodqDvRpL6DKoaSiUVNFUPJ\nc1GM+er2DEHwXW3IWqVuYeaNzPwggJMB/IGZzw9TORENIqJviWgxEd2kuf5/RDTb/ltERBulayOJ\n6Dv7L75Wq4ShOtfpzqvXgOrFWRgxVOkgaBOS73ZkBXdYMVSSC22Su3KdziKJcZ40QfWDL2liZiai\n9wD8yv69OGzFRJSG5aMxAMAyADOIaLKc8Y6Zr5HKXwkrzzaIqBmAMQB6wfLpmGXfuyFs+9UBccVQ\n1ZWzMGKo4qJQO1FZ5KQquEtFDJWkgjvq/IyCJJ8hsO0QZWYT0eEx6u4NYDEzf8/MuwC8CECfTs3C\ncAAT7OOBAN5j5vU2gXgPwKAYfShp5MpZVDdrKJH5zxCL4qBQ710OBS7EUGIcyAREBRE5RKW67cpr\nA2fhl4NbkK3DYXEF3xLRl0T0FRF9GaLutgB+ln4vs8/p2uoAoCOAD6LeW50he2IfeOCBznEQsdh3\n330BAJ07d3ad9xo8xUzzKTsEBuV+NkgWhfTcB4Cvv/7aIRbC92KfffbJEkPJPj/CEVCXuztXyH4l\nX331VV7rbtWqlXMs0tAm4fQqz/FCR0Lwm7VCiX0qgIMBDAbwWwBD7f/5xDAAk2yrq9AgolFENJOI\nZq5ZsybPXUoehxxyiHM8bNgw57isrAx33XWX81uVLZ9yyin4+OOPccUVVzjnFi5c6ArlIGPQoEH4\n+OOPsWjRIs8ySWHMmDFZ5wxnURwcc8wxeOWVV/Dmm29i3rx5ibfXsmVLhzCItMWnnnpqlhjq/fff\nx+zZswFYaVxff/31RFLvNmnSxDleunRpXur84YcfcO+997qcaidOnIhZs2YlQpzltUDdLCYNP6EX\nAQAzL4lZ93IA8pa2nX1Oh2EALlfuPVa59yP1JmZ+DMBjANCrV6/STVHmAcEhANkE4de//rVzrNuJ\n9+3rjsIihwvRQS1fKDRu3BgDBw7ElClTitK+QRXKyspwxhlnFKStdu3auZzy6tevjwYNGjj5TWRi\n0aRJE2chb9y4MYYM8ZNWx4c8x8Lk3g6D/fbbD3/84x+xbNkyAFb8qoYNG6JHjx55qd8PDRo0SLwN\nGX7EogURXet1kZn/HlD3DACdiKgjrMV/GIBz1EJ2KJGmAKZLp6cAuIOIxBc9EcDNAe1VO/jtsJNW\n9hUSqiKumDoUg8JARBuW85eIcPWlkFo339xtruHT46CUrKHSABrC5jCigpkriOgKWAt/GsB4Zp5P\nRGMBzGTmyXbRYQBeZEkTyszriehWWAQHAMYyc41LuORnzSAPumJPrFwhB4szqB0oKytzQtOL8SuS\nfZVCmPokraEKhUJbQ/m1tpKZx+ZSOTO/DeBt5dxo5fctHveOh+UIWGPh97FlAlEd8jv7QSUWxTTl\nNSgMysrKkMlksoiFnzVUofuXT+SaHzwOSolYVO8VqhqgtoqhDLGo+RBiKCIyYqiEUDKmswCOL1gv\nainCchbFnli5whCL2gfBWchchBFD5Rcl45RXE3UEpYbawll4eaAb1FzoFNylJIYynEV0FJY0GbhQ\nWxTcKgyxqPlIp9NYt26dyysbADZs2IBVq1YV3eghqV15IYlFKeksDBLGrl27PK/JC+pee+1ViO4k\nBtV5SPYvMaiZ2Lx5M3bv3o2tW7c6zmlbtmzBypUrsXjxYrRp06ao/ct3FjvhrX3aaafltV4/yOmV\nCwFDLIoIsWjq0lvKxOLOO+8sWJ+SwHXXXYcOHTrgxx9/xLHHHuuEQzCouWjVqhU2b97sCvXRqlUr\nNG7cGI0aNcJBBx1UlH4tW7YMc+bMweDBg/Nab7169bBs2TK0aNEir/X6IYmQKH4wxKKIEGxkp06d\nsq4JYtGzZ080atSooP3KN+rUqYPhw4cXuxsGBUSzZs2wceNGMDNatmzpnBM6i3322aco/Wrbti3a\ntk0mzFxS9ZYKapYwvJohjMyxuvtYGNRO6MxkxbmKioqiK7gNosMQiyLCT3FtlMAG1Rk6M1n5XKGV\nswa5wxCLEoUgFoazMKiO0JnJinMVFRWGWFRDGGJRRAiCoOMiDLEwqM4wYqiaB0MsShSGWBhUZxgx\nVM2DIRZFhCAEOoJgiIVBdUaQGMpwFtUPhryXOAyxMKiOSKVSTja6iooK55zhLKovDGdRRAgv1vPP\nPz/rmshdfc45WfmiDAxKHv/85z+d40ceeQSAxVmsX2+FnFu3bl1R+lUTUFZW5uQqL2i7BW/RwMHe\ne++NnTt3auPktG7d2vOagUGpY82aNVnn0uk0tm/fDgDo0qVLobtUY1BeXl4UiUOinAURDSKib4lo\nMRHd5FHmLCJaQETziegF6XyGiGbbf5N199YE1K1b1/PD+10zMKhuSKVS2L17N4DihPSuKSgrKytO\nSPSkKiaiNIAHAQwAsAzADCKazMwLpDKdYOXW7sPMG4iopVRFOTN3T6p/BgYGySGVSmXlWk+n007w\nzJoWSbk2IMkv1hvAYmb+npl3AXgRwBClzMUAHmTmDQDAzKsT7I+BgUGBoCMG6XTacBbVGEkSi7YA\nfpZ+L7PPyTgIwEFE9CkRfUZEg6Rr9Ylopn1eG/eXiEbZZWbqZKQGBgbFgU58asRQ1RvFVnCXAegE\n4FgA7QB8TESHMvNGAB2YeTkR7Q/gAyKax8xL5JuZ+TEAjwFAr169TDAlA4MSgY5YCKc8cWxQvZAk\nZ7EcQHvpdzv7nIxlACYz825m/gHAIljEA8y83P7/PYCPAByeYF8NDAzyCC9iIWB0FtUPSX6xGQA6\nEVFHIqoLYBgA1arpdVhcBYioOSyx1PdE1JSI6knn+wBYAAMDg2oBLzGUgOEsqh8SE0MxcwX9//bu\nPlaOqozj+PeXvpJCoKW1uaHElqSkQStwcyGXF02DUioSsYRItYmNmKIYfIEIKZJgiv/Ulxg1IcEq\nRAyvCgo3hFKroBAI0FtoS1ssvRQNpUArLQiCvJTHP+ZsO9zbMi13Z4fd+X2Szc6cmZ17zrO5++w5\nM3tGughYDowArouI9ZKuAvojoi9tmy1pA7ALuDQiXpJ0MvArSe+SJbQl+auozOzDrahn4WTRfko9\nZxERdwN3Dyq7MrccwCXpkd/nIWBmmXUzs/KMGzeON954A4D58+cD7D65DR6Gakd+x8ys6bq7uwFY\ntmwZN9xwAwBz587dvd09i/bjZGFmLeFhqPbmZGFmTeeroTqP3zEzawlfDdXenCzMrOn2dqtgD0O1\nNycLMytNfjjKw1Dtze+YmbWEexbtzcnCzEqTH47yOYv25mRhZi3hYaj25nfMzJpu8eLFdHV10dvb\nu7ssnywmTpxYRbVsGKqeotzMOlBvby9bt259T1m+NzF58uRWV8mGyT0LM2uJfM9i5Eh/T203ThZm\n1hK+Gqq9OVmYWUvkh6Hcs2g/ThZm1hLuWbQ3Jwsza4l8gtjbRIP24VZqspA0R9JGSQOSFu1jny9K\n2iBpvaSbcuULJG1KjwVl1tPMyuffVrS30gYOJY0ArgZOB7YAKyX15W+PKmk6cDlwSkTslPSRVD4B\n+AHQAwSwKr12Z1n1NbNyeeipvZWZ6k8EBiJic0S8BdwCnD1on4XA1Y0kEBHbUvkZwIqI2JG2rQDm\nlFhXMyuZk0V7KzNZHAE8m1vfksryjgaOlvSgpIclzTmA1yLpAkn9kvq3b9/exKqbWbN1dXVVXQUb\nhqoHEUcC04FZwJeAX0s6bH9fHBFLI6InInomTZpUUhXNrBlGjRpVdRVsGMpMFs8BR+bWp6SyvC1A\nX0S8HRHPAE+RJY/9ea2ZmbVImcliJTBd0jRJo4F5QN+gfe4g61UgaSLZsNRmYDkwW9J4SeOB2anM\nzMwqUNrVUBHxjqSLyD7kRwDXRcR6SVcB/RHRx56ksAHYBVwaES8BSPohWcIBuCoidpRVVzMze3/a\n271y21FPT0/09/dXXQ0zex+NH+N1yudOJ5C0KiJ6ivar+gS3mZm1AScLMzMr5GRhZmaFnCzMzKyQ\nJ5U3s5ZZunQpM2fOrLoa9gE4WZhZyyxcuLDqKtgH5GEoMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAz\ns0JOFmZmVsjJwszMCjlZmJlZoY6ZolzSduBfwzjERODfTapOp3BMhnJMhnJMhmqnmHw0IgrvS90x\nyWK4JPXvz5zudeKYDOWYDOWYDNWJMfEwlJmZFXKyMDOzQk4WeyytugIfQo7JUI7JUI7JUB0XE5+z\nMDOzQu5ZmJlZIScLMzMrVPtkIWmOpI2SBiQtqro+ZZJ0naRtktblyiZIWiFpU3oen8ol6ZcpLmsl\ndedesyDtv0nSgira0iySjpR0n6QNktZL+k4qr21cJI2V9KikNSkmi1P5NEmPpLbfKml0Kh+T1gfS\n9qm5Y12eyjdKOqOaFjWPpBGSHpd0V1qvT0wiorYPYATwNHAUMBpYAxxTdb1KbO+ngG5gXa7sx8Ci\ntLwI+FFaPhNYBgjoBR5J5ROAzel5fFoeX3XbhhGTLqA7LR8CPAUcU+e4pLYdnJZHAY+ktv4emJfK\nrwEuTMvfBK5Jy/OAW9PyMel/agwwLf2vjai6fcOMzSXATcBdab02Mal7z+JEYCAiNkfEW8AtwNkV\n16k0EXE/sGNQ8dnA9Wn5euALufLfReZh4DBJXcAZwIqI2BERO4EVwJzya1+OiHg+Ih5Ly68CTwJH\nUOO4pLa9llZHpUcApwG3pfLBMWnE6jbg05KUym+JiDcj4hlggOx/ri1JmgJ8DvhNWhc1ikndk8UR\nwLO59S2prE4mR8TzafkFYHJa3ldsOjZmaajgeLJv0rWOSxpuWQ1sI0t8TwMvR8Q7aZd8+3a3PW1/\nBTicDosJ8HPgMuDdtH44NYpJ3ZOF5UTWT67ltdSSDgZuB74bEf/Jb6tjXCJiV0QcB0wh++Y7o+Iq\nVUrSWcC2iFhVdV2qUvdk8RxwZG59SiqrkxfTMArpeVsq31dsOi5mkkaRJYobI+KPqbj2cQGIiJeB\n+4CTyIbcRqZN+fbtbnvafijwEp0Vk1OAz0v6J9lw9WnAL6hRTOqeLFYC09MVDaPJTkT1VVyn62FJ\nYQAAA0RJREFUVusDGlfuLADuzJV/JV390wu8koZllgOzJY1PVwjNTmVtKY0jXws8GRE/y22qbVwk\nTZJ0WFo+CDid7FzOfcC5abfBMWnE6lzg3tQb6wPmpSuDpgHTgUdb04rmiojLI2JKREwl+5y4NyLm\nU6eYVH2GveoH2dUtT5GNyV5RdX1KbuvNwPPA22RjpV8jG0f9K7AJ+AswIe0r4OoUlyeAntxxzic7\nMTcAfLXqdg0zJqeSDTGtBVanx5l1jgvwCeDxFJN1wJWp/CiyD7YB4A/AmFQ+Nq0PpO1H5Y51RYrV\nRuCzVbetSfGZxZ6roWoTE0/3YWZmheo+DGVmZvvBycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszBJJ\nr6XnqZK+3ORjf3/Q+kPNPL5Z2ZwszIaaChxQssj9indf3pMsIuLkA6yTWaWcLMyGWgJ8UtJqSRen\nSfV+ImlluofF1wEkzZL0gKQ+YEMqu0PSqnQfiAtS2RLgoHS8G1NZoxejdOx1kp6QdF7u2H+TdJuk\nf0i6Mf3aHElLlN1/Y62kn7Y8OlZLRd+GzOpoEfC9iDgLIH3ovxIRJ0gaAzwo6c9p327g45FNNw1w\nfkTsSNNkrJR0e0QsknRRZBPzDXYOcBxwLDAxveb+tO144GPAVuBB4BRJTwJzgRkREY1pOczK5p6F\nWbHZZPNBrSabvvxwsjl9AB7NJQqAb0taAzxMNmHcdN7fqcDNkc3y+iLwd+CE3LG3RMS7ZNOQTCWb\n6vp/wLWSzgFeH3brzPaDk4VZMQHfiojj0mNaRDR6Fv/dvZM0C/gMcFJEHEs2v9LYYfzdN3PLu4CR\nkd0b4USyG+qcBdwzjOOb7TcnC7OhXiW7xWrDcuDCNJU5ko6WNG4vrzsU2BkRr0uaQXYr0oa3G68f\n5AHgvHReZBLZrW/3OQtpuu/GoRFxN3Ax2fCVWel8zsJsqLXArjSc9Fuy+xZMBR5LJ5m3s+f2mXn3\nAN9I5xU2kg1FNSwF1kp6LLKprRv+RHaviDVks99eFhEvpGSzN4cAd0oaS9bjueSDNdHswHjWWTMz\nK+RhKDMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAr9H8BkRAnhng40AAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5d11d5e828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# Plot training accuracy over time\n",
    "plt.plot(range(len(train_acc_avg)), train_acc_avg, 'k-', label='Train Accuracy')\n",
    "plt.title('Avg Training Acc Over Past 50 Iterations')\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Training Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worthwhile to mention the motivation of limiting the sentence (or text) size.  In this example we limited the text size to 25 words.  This is a common practice with bag of words because it limits the effect of text length on the prediction.  You can imagine that if we find a word, “meeting” for example, that is predictive of a text being ham (not spam), then a spam message might get through by putting in many occurrences of that word at the end.  In fact, this is a common problem with imbalanced target data.  Imbalanced data might occur in this situation, since spam may be hard to find and ham may be easy to find.  Because of this fact, our vocabulary that we create might be heavily skewed toward words represented in the ham part of our data (more ham means more words are represented in ham than spam).  If we allow unlimited length of texts, then spammers might take advantage of this and create very long texts, which have a higher probability of triggering non-spam word factors in our logistic model.\n",
    "\n",
    "In the next section, we attempt to tackle this problem in a better way using the frequency of word occurrence to determine the values of the word embeddings."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
